Thursday, November 29, 2012

shell.function to download all files from any GITHUB GIST URL

a shell utility to download all files part of a GIST url at provided download path

usage e.g. $ ddl-gist https://gist.github.com/4165864 /etc/profile.d/

howTo install
$ curl -L -o /etc/profile.d/ddl-gist.sh https://gist.github.com/4165864

actually, if you had this utlity... the usage example would install the utlity... chicken-egg

# Usage Example: $ ddl-gist 'https://gist.github.com/4137843' ~/Downloads/gists
# save the gist files at that URL in ~/Downloads/gists
##
ddl_gist(){
if [ $# -ne 2 ];
then
echo 'Failed. Syntax: $> ddl-gist GITHUB_GIST_URL DOWNLOAD_PATH'
return
fi
gist_url=$1
download_path=$2
echo '[*] Getting all GIST File URLs from '$gist_url
gists=`curl -ksL -H 'User-Agent: Mozilla/5.0' $gist_url | grep '<a\ .*href=".*/raw/' | sed 's/.*a\ .*href="//' | sed 's/".*//'`
echo '[*] Downloading all files'
for lines in `echo $gists | xargs -L1`;
do
if [ ! -z $lines ];
then
echo $lines
gistfile=`echo $lines | sed 's/.*\///'`
save_as=$download_path"/"$gistfile
echo "Downloading URL: https://gist.github.com"$lines
echo "to "$save_as"....."
wget -c -O $save_as "https://gist.github.com"$lines
fi
done
}
view raw gistfile1.sh hosted with ❤ by GitHub

Tuesday, November 27, 2012

auto-download all media from RSS - DefCon20 Talk Videos

a shell utility code for download all media present as enclosures in any RSS

eg. usage to download all slides video of DefCon20
$ ddl-rss-media https://www.defcon.org/podcast/defcon-20-slides.rss

to install it
$ sudo curl -L -o /etc/profile.d/ddl-rss-media.sh https://gist.github.com/4156364#file_gistfile1.sh

#####
## $ ddl-rss-media https://www.defcon.org/podcast/defcon-20-slides.rss
#####
ddl-rss-media(){
# ddl-rss-media RSS_LINK {would download all media enclosed at current dir}
enclosures=`curl -k -s -L $@ | cat | grep enclosure | sed 's/.*enclosure\s*url="//' | sed 's/".*//'`
for url in `echo $enclosures | xargs -L1`;
do
if [ ! -z $url ];
then
filename=`echo $url | sed 's/?.*//' | sed 's/.*\///'`
echo "Downloading $filename..."
wget -c -O $filename $url
fi
done
}
view raw gistfile1.sh hosted with ❤ by GitHub

auto-download all videos of any Event at Confreaks.com

Auto-Download all smallest size Videos of any provided Event in Confreaks.com

to install it either run
$ sudo curl -L -o /etc/profile.d/ddl-confreaks.sh https://gist.github.com/raw/4137843/e6c6c93ef1c4cbf4cb735de821b74fcec467c170/gistfile1.sh

or copy+paste the shell function below in your shell-load-conf file.

suppose the event resides at URL:
http://confreaks.com/events/rubyconf2012

and you just need to run
$ ddl-confreaks rubyconf2012 ~/Downloads

and it will download all Talk Videos from Ruby Conference 2012 at your ~/Downloads directory...

would download all small Videos found at all talks under that Event Page

# Usage Example: $ ddl-confreaks rubyconf2012 ~/Downloads/
# save it as /etc/profiles/ddl.confreaks.sh
##
## currently it checks for lowest resolution video mostly {640x360} and downloads it
function ddl-confreaks(){
if [ $# -ne 2 ];
then
echo 'Failed. Syntax: $> ddl-confreaks EVENT_NAME DOWNLOAD_PATH'
return
fi
event=$1
download_path=$2
event_url="http://confreaks.com/events/$event"
echo '[*] Getting all talk urls from the Event Page'
talk_urls=`curl -s $event_url | grep --color -A1 "title" | grep href | sed 's/.*href="/http:\/\/confreaks\.com/' | sed 's/".*//'`
echo '[*] Getting all MP4 Video URLs from the Talk Pages'
for lines in `echo $talk_urls | xargs -L1`;
do
xmp4_url=`echo $lines | xargs curl -s | grep 'application/x-mp4' | tail -1 | sed 's/.*href="//' | sed 's/".*//'`
if [ ! -z $xmp4_url ];
then
echo "Scanned: "$lines
mp4file=`echo $xmp4_url | sed 's/.*\///' | sed 's/\?.*//'`
save_as=$download_path"/"$mp4file
echo "Downloading URL: "$xmp4_url
echo "to "$save_as"....."
wget -c -O $save_as $xmp4_url
fi
done
}
view raw gistfile1.sh hosted with ❤ by GitHub