|
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
|
8518
|
18.65 %
|
|
|
MacOS
|
6056
|
13.26 %
|
|
|
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
|
2341
|
5.13 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
|
2140
|
4.69 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
|
2013
|
4.41 %
|
|
|
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
|
1726
|
3.78 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
|
1567
|
3.43 %
|
|
|
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
|
1503
|
3.29 %
|
|
|
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
|
1481
|
3.24 %
|
|
|
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
1268
|
2.78 %
|
|
|
WindowsXP
|
1169
|
2.56 %
|
|
|
WindowsNT
|
1018
|
2.23 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
|
859
|
1.88 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
|
802
|
1.76 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
|
767
|
1.68 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
|
674
|
1.48 %
|
|
|
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
|
530
|
1.16 %
|
|
|
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
|
495
|
1.08 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
|
470
|
1.03 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
|
398
|
0.87 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html)
|
362
|
0.79 %
|
|
|
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
355
|
0.78 %
|
|
|
Linux
|
339
|
0.74 %
|
|
|
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
|
326
|
0.71 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
|
292
|
0.64 %
|
|
|
DoCoMo2.1; +http:
|
289
|
0.63 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
|
288
|
0.63 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
|
284
|
0.62 %
|
|
|
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
|
270
|
0.59 %
|
|
|
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
|
260
|
0.57 %
|
|
|
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
|
255
|
0.56 %
|
|
|
findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
228
|
0.50 %
|
|
|
psbot/0.1 (+http://www.picsearch.com/bot.html)
|
222
|
0.49 %
|
|
|
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
|
218
|
0.48 %
|
|
|
Java/1.6.0_04
|
208
|
0.46 %
|
|
|
|
207
|
0.45 %
|
|
|
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
|
202
|
0.44 %
|
|
|
Go-http-client/1.1
|
168
|
0.37 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
|
165
|
0.36 %
|
|
|
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
|
134
|
0.29 %
|
|
|
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
|
126
|
0.28 %
|
|
|
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
|
116
|
0.25 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
|
107
|
0.23 %
|
|
|
CCBot/2.0 (https://commoncrawl.org/faq/)
|
106
|
0.23 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
|
105
|
0.23 %
|
|
|
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
|
104
|
0.23 %
|
|
|
ltx71 - (http://ltx71.com/)
|
101
|
0.22 %
|
|
|
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
92
|
0.20 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
|
84
|
0.18 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
|
78
|
0.17 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
|
73
|
0.16 %
|
|
|
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
|
67
|
0.15 %
|
|
|
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
|
66
|
0.14 %
|
|
|
DoCoMomobile goo; +http:
|
65
|
0.14 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
|
65
|
0.14 %
|
|
|
Windows98
|
64
|
0.14 %
|
|
|
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
|
60
|
0.13 %
|
|
|
Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
|
56
|
0.12 %
|
|
|
Mozilla/4.0+(compatible;+Robot/1.0)
|
56
|
0.12 %
|
|
|
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
|
51
|
0.11 %
|
|
|
Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
|
49
|
0.11 %
|
|
|
Mozilla/5.0 (compatible; DnyzBot/1.0)
|
47
|
0.10 %
|
|
|
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
47
|
0.10 %
|
|
|
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.ninjal.ac.jp/corpus_center/ulc/crawl-en)
|
45
|
0.10 %
|
|
|
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
|
43
|
0.09 %
|
|
|
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
|
43
|
0.09 %
|
|
|
python-requests/2.31.0
|
43
|
0.09 %
|
|
|
LinkWalker/2.0
|
43
|
0.09 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
|
42
|
0.09 %
|
|
|
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
|
42
|
0.09 %
|
|
|
CCBot/2.0 (http://commoncrawl.org/faq/)
|
42
|
0.09 %
|
|
|
TsolCrawler/Nutch-1.4 (http://www.toshiba-sol.co.jp/info/140100.htm; 'tsol-itc-crawler at toshiba-sol dot co dot jp')
|
41
|
0.09 %
|
|
|
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
|
39
|
0.09 %
|
|
|
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
|
37
|
0.08 %
|
|
|
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
36
|
0.08 %
|
|
|
claudebot
|
35
|
0.08 %
|
|
|
Wotbox/2.01 (+http://www.wotbox.com/bot/)
|
33
|
0.07 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
|
33
|
0.07 %
|
|
|
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
32
|
0.07 %
|
|
|
Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)
|
32
|
0.07 %
|
|
|
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
|
31
|
0.07 %
|
|
|
BUbiNG (+http://law.di.unimi.it/BUbiNG.html#wc)
|
31
|
0.07 %
|
|
|
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
|
31
|
0.07 %
|
|
|
Tobot/Nutch-1.4 ('tobot at eel dot rdc dot toshiba dot co dot jp')
|
31
|
0.07 %
|
|
|
MacOSPowerPC
|
30
|
0.07 %
|
|
|
Mozilla/5.0 (compatible; Clarabot/1.4; +http://www.clarabot.info/bots)
|
29
|
0.06 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
|
29
|
0.06 %
|
|
|
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
28
|
0.06 %
|
|
|
Lynx/2.8.5rel.1 libwww-FM/2.15FC SSL-MM/1.4.1c OpenSSL/0.9.7e-dev
|
28
|
0.06 %
|
|
|
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
|
27
|
0.06 %
|
|
|
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
|
26
|
0.06 %
|
|
|
Nutch12/Nutch-1.2
|
25
|
0.05 %
|
|
|
MauiBot (crawler.feedback+wc@gmail.com)
|
25
|
0.05 %
|
|
|
MADBbot/0.1 (Gathering webpages for data analytics; https://madb.zapto.org/bot.html; ma-db-crawl@googlegroups.com)
|
25
|
0.05 %
|
|
|
DoCoMo1.0; http:
|
25
|
0.05 %
|
|
|
Windows2000
|
24
|
0.05 %
|
|
|
DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/1alpha.jp)
|
24
|
0.05 %
|
|
|
Scrapy/2.11.1 (+https://scrapy.org)
|
24
|
0.05 %
|
|
|
Owler (ows.eu/owler)
|
23
|
0.05 %
|
|
|
Mozilla/5.0 (compatible; Adsbot/3.1)
|
23
|
0.05 %
|
|
|
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
|
21
|
0.05 %
|
|
|
Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
|
21
|
0.05 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.1) Gecko/20100101 Firefox/26.0
|
21
|
0.05 %
|
|
|
SafeDNSBot (https://www.safedns.com/searchbot)
|
20
|
0.04 %
|
|
|
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
|
19
|
0.04 %
|
|
|
Java/1.6.0_22
|
19
|
0.04 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
|
18
|
0.04 %
|
|
|
Barkrowler experimental crawler - 0.4.7
|
18
|
0.04 %
|
|
|
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
|
16
|
0.04 %
|
|
|
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.ninjal.ac.jp/corpus_center/ulc/crawl-en.html)
|
16
|
0.04 %
|
|
|
Mozilla/4.0+(compatible;+Robot/1.2 CL)
|
16
|
0.04 %
|
|
|
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
|
16
|
0.04 %
|
|
|
Java/1.4.1_04
|
15
|
0.03 %
|
|
|
Mozilla/5.0+(compatible;+Robot/1.2)
|
15
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)
|
15
|
0.03 %
|
|
|
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
|
15
|
0.03 %
|
|
|
HD nutch agent/Nutch-1.1 (Think)
|
15
|
0.03 %
|
|
|
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
|
15
|
0.03 %
|
|
|
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
14
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
|
14
|
0.03 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Firefox/24.0
|
14
|
0.03 %
|
|
|
Java/1.7.0_79
|
14
|
0.03 %
|
|
|
TosCrawler/Nutch-1.4 (http://www.toshiba.co.jp/rdc/about/crawl_info.htm; 'Rdc-crawler at ml dot toshiba dot co dot jp')
|
14
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; Dataprovider.com;)
|
14
|
0.03 %
|
|
|
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
|
14
|
0.03 %
|
|
|
netEstate NE Crawler (+http://www.website-datenbank.de/)
|
14
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
|
14
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
|
13
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
|
13
|
0.03 %
|
|
|
Mozilla/5.0 (Windows NT 6.2; WOW64)
|
13
|
0.03 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
|
13
|
0.03 %
|
|
|
IS Alpha/Nutch-1.1
|
13
|
0.03 %
|
|
|
Mozilla/2.0 (compatible; crw)
|
13
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; tracemyfile/1.0)
|
12
|
0.03 %
|
|
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729
|
12
|
0.03 %
|
|
|
Scrapy/2.6.1 (+https://scrapy.org)
|
12
|
0.03 %
|
|
|
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
12
|
0.03 %
|
|
|
Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0)
|
12
|
0.03 %
|
|
|
ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
|
12
|
0.03 %
|
|
|
Python-urllib/3.10
|
12
|
0.03 %
|
|
|
siclab (cboc-test@lab.ntt.co.jp)
|
12
|
0.03 %
|
|
|
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
|
12
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
|
12
|
0.03 %
|
|
|
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
|
11
|
0.02 %
|
|
|
serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)
|
11
|
0.02 %
|
|
|
Barkrowler/0.9 (+https://babbar.tech/crawler)
|
11
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
|
10
|
0.02 %
|
|
|
Barkrowler/0.7 (+http://www.exensa.com/crawl)
|
10
|
0.02 %
|
|
|
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
|
10
|
0.02 %
|
|
|
ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
|
10
|
0.02 %
|
|
|
TsolCrawler/Nutch-1.4
|
10
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
|
10
|
0.02 %
|
|
|
WindowsNT 6.2
|
10
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
|
10
|
0.02 %
|
|
|
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
|
10
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
|
9
|
0.02 %
|
|
|
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
|
9
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/14.0.1
|
9
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0
|
9
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
|
9
|
0.02 %
|
|
|
EC2LinkFinder
|
9
|
0.02 %
|
|
|
DoCoMomobile goo;+http:
|
9
|
0.02 %
|
|
|
AgentName/0.1libwww-perl/6.02
|
8
|
0.02 %
|
|
|
HTTPClient/1.0 (2.8.3, ruby 2.7.4 (2021-07-07))
|
8
|
0.02 %
|
|
|
pflab (co2h2onacl@gmail.com)
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
|
8
|
0.02 %
|
|
|
FriendlyCrawler/Nutch-1.20-SNAPSHOT
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
|
8
|
0.02 %
|
|
|
Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)
|
8
|
0.02 %
|
|
|
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
|
8
|
0.02 %
|
|
|
TosCrawler/Nutch-1.5.1 (http://www.toshiba.co.jp/rdc/about/crawl_info.htm; 'Rdc-crawler at ml dot toshiba dot co dot jp')
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
|
8
|
0.02 %
|
|
|
Jakarta Commons-HttpClient/3.1
|
8
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0
|
7
|
0.02 %
|
|
|
Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
|
7
|
0.02 %
|
|
|
Windows32
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
|
7
|
0.02 %
|
|
|
My Nutch Spider/Nutch-1.10
|
7
|
0.02 %
|
|
|
Toshiba Digital Solution TDSL/Nutch-1.8
|
7
|
0.02 %
|
|
|
User-Agent: Mozilla/4.0 (compatible; MSIE 6.11; WindowsNT 5.1
|
7
|
0.02 %
|
|
|
Mozilla/5.0
|
7
|
0.02 %
|
|
|
LSSRocketCrawler/1.0 LightspeedSystems
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (compatible; Cincraw/1.0; +http://cincrawdata.net/bot/)
|
7
|
0.02 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0
|
6
|
0.01 %
|
|
|
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
|
6
|
0.01 %
|
|
|
GeoHasher/Nutch-1.0 (GeoHasher Web Search Engine; geohasher.gotdns.org; geo_hasher at yahoo * com)
|
6
|
0.01 %
|
|
|
TosCrawler/Nutch-1.6 (http://www.toshiba.co.jp/rdc/about/crawl_info_en.htm; 'Rdc-crawler at ml dot toshiba dot co dot jp')
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us; rv:1.9.2.3) Gecko/20100401 YFF35 Firefox/3.6.3
|
6
|
0.01 %
|
|
|
WindowsNT 6.1
|
6
|
0.01 %
|
|
|
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
|
6
|
0.01 %
|
|
|
GTWek
|
6
|
0.01 %
|
|
|
Ruby
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
|
6
|
0.01 %
|
|
|
ICC-Crawler/3.0 (Mozilla-compatible; ; https://ucri.nict.go.jp/en/icccrawler.html)
|
6
|
0.01 %
|
|
|
netEstate NE Crawler (+http://www.sengine.info/)
|
6
|
0.01 %
|
|
|
user-agent
|
6
|
0.01 %
|
|
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
|
5
|
0.01 %
|
|
|
IE/4.0
|
5
|
0.01 %
|
|
|
Turnitin (https://bit.ly/2UvnfoQ)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
|
5
|
0.01 %
|
|
|
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
|
5
|
0.01 %
|
|
|
Nutch/Nutch-2.0 (Nutch Crawler)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:54.0) Gecko/20100101 Firefox/54.0
|
5
|
0.01 %
|
|
|
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100
|
5
|
0.01 %
|
|
|
Baiduspider+(+http://www.baidu.jp/spider/)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
|
5
|
0.01 %
|
|
|
python-requests/2.18.4
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
|
5
|
0.01 %
|
|
|
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
|
5
|
0.01 %
|
|
|
mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; ExtLinksBot/1.5; +https://extlinks.com/Bot.html)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)
|
5
|
0.01 %
|
|
|
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0
|
4
|
0.01 %
|
|
|
Scrapy/1.7.2 (+https://scrapy.org)
|
4
|
0.01 %
|
|
|
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
|
4
|
0.01 %
|
|
|
Scrapy/2.8.0 (+https://scrapy.org)
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
|
4
|
0.01 %
|
|
|
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; Robo/1.0b; http://potaru.com/robo.html)/Nutch-1.2
|
4
|
0.01 %
|
|
|
SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
|
4
|
0.01 %
|
|
|
Java/11.0.10
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160721-2308 +https://www.domaincodex.com)
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b1) Gecko/20091014 Firefox/3.6b1 GTB5
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
|
4
|
0.01 %
|
|
|
Mozilla/6.0 (compatible)
|
4
|
0.01 %
|
|
|
GSLFbot
|
4
|
0.01 %
|
|
|
serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
|
4
|
0.01 %
|
|
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0
|
4
|
0.01 %
|
|
|
Stratagems Kumo
|
4
|
0.01 %
|
|
|
INETDEX-BOT/1.5 (Mozilla/5.0; https://inetdex.com/bot.html)
|
4
|
0.01 %
|
|
|
OpenAcoon v4.10.5 (www.openacoon.de)
|
4
|
0.01 %
|
|