regex for robots.txt curl output doesn't work

catalin :

curl -s http://hostname/robots.txt

Output needed to parse:

User-Agent: *
Disallow: /
User-Agent: Mediapartners-Google*

I need to take User-Agent: * and check it's starting with User-agent and ending with * Tried curl -s "http://hostname" | grep -i "^User\-Agent:\ \*$" but I get nothing.

catalin :

solution is to redirect the output to another file and convert it to unix file.

curl -s http://hostname/robots.txt > /tmp/some_file.txt
dos2unix /tmp/some_file.txt
cat /tmp/some_file.txt | grep -i "^User\-Agent\:\ \*$"