Hello,
Since is seems almost to contact FB concerning about from the Crawler (no reply from noc, abuse, and customer support is closing tickets without looking) we are started to get a bit annoyed about the issue we are facing.
Since a few days the
De User-Agent from the http request is : facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
We also checked that all IP addresses are originating from your datacenters as described on : https://developers.facebook.com/docs/sharing/webmasters/crawler/
Due to the crazy amount of request the load on our server increased from 0.5 to 12-15 average, and websites hosted on this server were almost unusable.
Because of that we needed to add a ratelimit on the user-agent facebookexternalhit/1.1
(+http://www.facebook.com/externalhit_uatext.php) but this has side effects, that when people are sharing a post, the crawler cannot always fetch the information from the webserver on our server, because it's being limited by the server.
To give some idea about the amount of requests :
root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.5.gz 83161 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.4.gz 197321 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.3.gz 137907 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.2.gz 84284 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.1.gz 62772
This is from the last 5 days, where we started limiting about 3days ago, but when disable the ratelimit again, the load of the server is back to 15-20 in 2 minuts.
Please take this message seriously. i've tried almost anything to find a solution.
It does not help there is no point of contact for this API / Crawler,the information site is a great place to post contact info: https://developers.facebook.com/docs/sharing/webmasters/crawler/
I can share the info about the website, but i rather do that in DM .
Can you please help for this issue?
We are also experiencing issues with facebookexternalhit/1.1 starting May 25
We have fully blocked the user-agent after confirming all requests are coming from Facebook IP addresses. We received 40k requests starting May 25, 2024.
The user-agent "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" is apparently from Facebook's "Sharing Debugger" tool or from a "Social Plugins" rather than it's FacebookBot crawler.
And, furthermore, it is very suspicious to us that shortly after these thousands of requests began, we also started experiencing a card testing attack. However, the card testing has continued despite the block on facebookexternalhit/1.1 requests
Similar to above, my server suffered several tens of thousands of requests from bot facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Now I have to block this bot. I wonder if you have any solution or information about this facebookexternalhit/1.1. Why does it continuously send requests to the server?
I've filed a bugreport for this, hopefully they are going to look into this issue.
If you have any information, please leave a message here to let me know. Thank you.
We also experience this and Facebook is doing NOTHING against it. Obviously it would be very easy: if a human being is sharing a link on Facebook, the useragent should be a different one compared to those made through an API. This way it would be so easy to distinguish these two cases.
As of today it started again after being quiet for almost a week. getting really tired of this.
