exploit the possibilities
Home Files News &[SERVICES_TAB]About Contact Add New

Applebot Incorrect Robots.txt Interpretation

Applebot Incorrect Robots.txt Interpretation
Posted Jul 7, 2020
Authored by David Coomber

Applebot/0.1 does not fully obey robots.txt as it interprets allow entries for Googlebot as implied permission for Applebot.

tags | advisory
SHA-256 | 78ae053c4168117e295b49f3d21f45583932d75f34e82ed990e26f77540f353c

Applebot Incorrect Robots.txt Interpretation

Change Mirror Download
Applebot - Not Obeying robots.txt
https://www.info-sec.ca/advisories/Applebot.html

Overview
"Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot. It respects customary robots.txt rules and robots meta tags, and it originates in the 17.0.0.0 net block."

(https://support.apple.com/en-us/HT204683)

Issue
Applebot/0.1 does not fully obey robots.txt as it interprets allow entries for Googlebot as implied permission for Applebot, illustrated by the following example:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Impact
This scenario presents potential privacy implications for site owners who may want their content made available to Google but not Apple.

Timeline
September 19, 2019 - Provided the details to Apple via product-security@apple.com
September 19 , 2019 - Apple sent an auto acknowledgment
October 1, 2019 - Apple responded stating that they are investigating
October 29, 2019 - Asked for a status update
November 4, 2019 - Apple responded stating that they do not see any security implications
July 7, 2020 - Published an advisory to document the issue

Solution
If you do not want Apple to index your site, explicitly restrict Applebot in robots.txt using the following example:

User-agent: Applebot
Disallow: /
Login or Register to add favorites

File Archive:

April 2024

  • Su
  • Mo
  • Tu
  • We
  • Th
  • Fr
  • Sa
  • 1
    Apr 1st
    10 Files
  • 2
    Apr 2nd
    26 Files
  • 3
    Apr 3rd
    40 Files
  • 4
    Apr 4th
    6 Files
  • 5
    Apr 5th
    26 Files
  • 6
    Apr 6th
    0 Files
  • 7
    Apr 7th
    0 Files
  • 8
    Apr 8th
    22 Files
  • 9
    Apr 9th
    14 Files
  • 10
    Apr 10th
    10 Files
  • 11
    Apr 11th
    13 Files
  • 12
    Apr 12th
    14 Files
  • 13
    Apr 13th
    0 Files
  • 14
    Apr 14th
    0 Files
  • 15
    Apr 15th
    30 Files
  • 16
    Apr 16th
    10 Files
  • 17
    Apr 17th
    22 Files
  • 18
    Apr 18th
    45 Files
  • 19
    Apr 19th
    0 Files
  • 20
    Apr 20th
    0 Files
  • 21
    Apr 21st
    0 Files
  • 22
    Apr 22nd
    0 Files
  • 23
    Apr 23rd
    0 Files
  • 24
    Apr 24th
    0 Files
  • 25
    Apr 25th
    0 Files
  • 26
    Apr 26th
    0 Files
  • 27
    Apr 27th
    0 Files
  • 28
    Apr 28th
    0 Files
  • 29
    Apr 29th
    0 Files
  • 30
    Apr 30th
    0 Files

Top Authors In Last 30 Days

File Tags

Systems

packet storm

© 2022 Packet Storm. All rights reserved.

Services
Security Services
Hosting By
Rokasec
close