검색엔진에서 방문하는 다양한 bot

현재 접속자 정보에 로봇 모두 제외시키기

접속정보에 보면 로봇이 엄청나게 많이 붙어있는데, 이것이 보기 싫은 분을 위한 코드입니다.

이 코드를 사용하면, 봇은 접속할 수 있으나, 흔적이 전혀 남지 않으므로, 사용에 유의하시기 바랍니다.

/lib/common.lib.php 소스를 열어보시면 

약 2800 라인 정도에  function run() 이라는 함수가 있습니다.

함수시작부분부터 약 15라인 정도 내려가면 else{ 바로 뒤에

 $tmp_sql = " insert into {$g5['login_table']} ( lo_ip, mb_id, lo_datetime, lo_location, lo_url ) values ( '{$_SERVER['REMOTE_ADDR']}', '{$member['mb_id']}', '".G5_TIME_YMDHIS."', '{$g5['lo_location']}',  '{$g5['lo_url']}' ) ";

                sql_query($tmp_sql, FALSE);

이부분이 있는데, 이부분이 접속정보를 DB에 저장하는 부분입니다.

 $agent = $_SERVER['HTTP_USER_AGENT'];

            if( strpos($agent,'bot') === false && strpos($agent,'Google') === false && strpos($agent,'Yeti') === false && strpos($agent,'NAVER') === false) {// 봇이 아니면 삽입

              $tmp_sql = " insert into {$g5['login_table']} ( lo_ip, mb_id, lo_datetime, lo_location, lo_url ) values ( '{$_SERVER['REMOTE_ADDR']}', '{$member['mb_id']}', '".G5_TIME_YMDHIS."', '{$g5['lo_location']}',  '{$g5['lo_url']}' ) ";

                sql_query($tmp_sql, FALSE);

            }

파란색 부분이 추가된 부분입니다. 

bot들의 종류가 많고, 이름이 다양해서, 조건이 좀 깁니다. 

추가로 발견되는 봇이 있으면, if 조건문에 추가하면 됩니다.

검색엔진에서 방문하는 다양한 bot 을 정리해봤습니다.

1~0
1Noonbot
360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
360Spider-Image
360Spider-Video

A
(compatible; AdsBot-Naver/1.0; +http://searchad.naver.com)
AhrefsBot/5.0
AhrefsBot/5.1
(compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
(compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

B
(compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
BaiduSpider
(compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
(Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx)
BLEXBot/1.0
BusinessBot: Nathan@lead-caddy.com

C
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
Contacts-Crawler/0.2 (+http://www.scrapinghub.com)

D
(compatible; Dataprovider.com;)
DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
(compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Digincore bot
Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
(compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
DotBot/1.1
DuckDuckGo-Favicons-Bot/1.0

E
empas
Exabot/3.0

F


G
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot-Image
(compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

H
360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

I
ia_archiver
IDBot/1.1
linkdexbot/2.2

J


K
kisaBot v1.0
KisaBotD/2.0

L


M
(compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
(compatible, Linux x86_64; Mail.RU_Bot/Fast/2.0; +//go.mail.ru/help/robots)
(compatible, Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
(compatible, Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
meanpathbot/1.0
MediavBot/1.0 (+http://www.mediav.com/)
(compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
MJ12bot/v1.4.4
MJ12bot/v1.4.5
MJ12bot/v1.4.6
(compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
(compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
MSNBot
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

N
Naverbot
NeumobBot/0.2.16 (+http://www.neumob.com/bot)

O
oBot/2.3.1

P
(compatible; PageAnalyzer/1.5;)
PagesInventory (robot +http://www.pagesinventory.com)
Perviibot/1.0
(compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
PrivacyAwareBot/1.1

Q
 

R
RankvalBot/1.0
roboto
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
RSNBOT CONTACT US 031-924-2025

S
SafeDNSBot (https://www.safedns.com/searchbot)
semanticbot
semanticbot (info@semanticaudience.com)
SemrushBot/0.98~bl
SemrushBot/0.99~bl
SemrushBot/1~bl
SemrushBot/1.1~bl
(compatible; seoscanners.net/1; +spider@seoscanners.net)
(compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
(compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Slurp
(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
spbot/5.0
spbot/5.0.2
spbot/5.0.3
SurveyBot/2.3 (DomainTools)

T
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)

U
Uptimebot/0.1.73
Uptimebot/0.2.42
Uptimebot/0.2.43
Uptimebot/1.0
UXCrawlerBot

V
Virusdie crawler/3.0

W
WBSearchBot/1.1
WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1
wisebot
WoTBoT
Wotbox/2.01 (+http://www.wotbox.com/bot/)

X
XoviBot/2.0

Y
(compatible; YandexBot/3.0; +http://yandex.com/bots)
Yahoo-MMCrawler
(compatible; Yeti/1.1; +http://naver.me/bot)
(compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
YisouSpider

Z
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)

출처: http://www.happyjung.com/lecture/2619