what you don't know can hurt you
Home Files News &[SERVICES_TAB]About Contact Add New

Applebot Incorrect Robots.txt Interpretation

Applebot Incorrect Robots.txt Interpretation
Posted Jul 7, 2020
Authored by David Coomber

Applebot/0.1 does not fully obey robots.txt as it interprets allow entries for Googlebot as implied permission for Applebot.

tags | advisory
SHA-256 | 78ae053c4168117e295b49f3d21f45583932d75f34e82ed990e26f77540f353c

Applebot Incorrect Robots.txt Interpretation

Change Mirror Download
Applebot - Not Obeying robots.txt
https://www.info-sec.ca/advisories/Applebot.html

Overview
"Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot. It respects customary robots.txt rules and robots meta tags, and it originates in the 17.0.0.0 net block."

(https://support.apple.com/en-us/HT204683)

Issue
Applebot/0.1 does not fully obey robots.txt as it interprets allow entries for Googlebot as implied permission for Applebot, illustrated by the following example:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Impact
This scenario presents potential privacy implications for site owners who may want their content made available to Google but not Apple.

Timeline
September 19, 2019 - Provided the details to Apple via product-security@apple.com
September 19 , 2019 - Apple sent an auto acknowledgment
October 1, 2019 - Apple responded stating that they are investigating
October 29, 2019 - Asked for a status update
November 4, 2019 - Apple responded stating that they do not see any security implications
July 7, 2020 - Published an advisory to document the issue

Solution
If you do not want Apple to index your site, explicitly restrict Applebot in robots.txt using the following example:

User-agent: Applebot
Disallow: /
Login or Register to add favorites

File Archive:

July 2024

  • Su
  • Mo
  • Tu
  • We
  • Th
  • Fr
  • Sa
  • 1
    Jul 1st
    27 Files
  • 2
    Jul 2nd
    10 Files
  • 3
    Jul 3rd
    35 Files
  • 4
    Jul 4th
    27 Files
  • 5
    Jul 5th
    18 Files
  • 6
    Jul 6th
    0 Files
  • 7
    Jul 7th
    0 Files
  • 8
    Jul 8th
    28 Files
  • 9
    Jul 9th
    44 Files
  • 10
    Jul 10th
    24 Files
  • 11
    Jul 11th
    25 Files
  • 12
    Jul 12th
    11 Files
  • 13
    Jul 13th
    0 Files
  • 14
    Jul 14th
    0 Files
  • 15
    Jul 15th
    28 Files
  • 16
    Jul 16th
    6 Files
  • 17
    Jul 17th
    34 Files
  • 18
    Jul 18th
    6 Files
  • 19
    Jul 19th
    34 Files
  • 20
    Jul 20th
    0 Files
  • 21
    Jul 21st
    0 Files
  • 22
    Jul 22nd
    19 Files
  • 23
    Jul 23rd
    17 Files
  • 24
    Jul 24th
    47 Files
  • 25
    Jul 25th
    0 Files
  • 26
    Jul 26th
    0 Files
  • 27
    Jul 27th
    0 Files
  • 28
    Jul 28th
    0 Files
  • 29
    Jul 29th
    0 Files
  • 30
    Jul 30th
    0 Files
  • 31
    Jul 31st
    0 Files

Top Authors In Last 30 Days

File Tags

Systems

packet storm

© 2022 Packet Storm. All rights reserved.

Services
Security Services
Hosting By
Rokasec
close