This is something I've started doing a lot these days out of need. First what I mean by hacking is figuring out how stuff works i.e. how the web client (the web browser) and web server (Apache, IIS, Tomcat) interact. There are a few tools freely available which can go a long in elucidating how stuff works. A lot of them are simple plugins for Firefox(LiveHTTP headers, Tamper Data, Firebug( or other web browsers. Others are standalone applications like web proxies (Scarab) or network interface sniffers (Wireshark, tcpdump).
The most obvious of hacking a website is simulating a web browser such as Firefox. In my case the language of choice to do that is Python. At the place I'work we've got a browser class written in Python which uses the built-in httplib, urllib and urlparse modules.
It also handles session cookies by storing them in a dict. The code is not fully RFC compliant but does the job for most websites.
By using the browser module we can try to simulate all post, get requests that go from a web browser like Firefox towards a web server. Obviously any client side script that usually runs in a compliant web browser needs to be simulated in Python code if it is found to be necessary in the client-server communication.
Simulating a web browser in Python is fairly interesting in itself because it does open your eyes to many vulnerabilities in server-client side scripts.
The really interesting stuff though is when web site designers try to stop us from accessing their web site programatically. Ways of accomplishing this usually involve the test which are trivial for humans but hard for machines (e.g. captcha codes).
...
Saturday 10 November 2007
Friday 5 October 2007
Starting my first job
Ok so I've started my first job. Stuff I'm doing for now is web development with python + ajax.
Tuesday 3 July 2007
My mrxvt modification
From the original mrxvt, I've kept and am currently working on what I believe to be important. Noticeably, support for scalable fonts (in particular Microsoft True Type fonts) which is provided by the Xft library and full support for UTF-8. Drawing UTF-8 encoded strings is not hard as it simply involves calling XftDrawStringUtf8. What is harder is to make sure that internally, strings are processed as potentially multi-byte per character strings at all times.
What is broken for now is
1) selection in mrxvt of utf-8 data, the glyphs just disappear once selected. Well actually only the first one seems to be copied and can only be pasted once in mrxvt
2) backspace doesn't work properly
For example in python I do:
print u"\u0540\u0539\u0538"
which prints out 3 characters:
ՀԹԸ
but when I select those 3 chars all that remains is:
Հ
and when I paste it in the terminal then it is not possible to delete the first character
I'll complete this post as I go along...
What is broken for now is
1) selection in mrxvt of utf-8 data, the glyphs just disappear once selected. Well actually only the first one seems to be copied and can only be pasted once in mrxvt
2) backspace doesn't work properly
For example in python I do:
print u"\u0540\u0539\u0538"
which prints out 3 characters:
ՀԹԸ
but when I select those 3 chars all that remains is:
Հ
and when I paste it in the terminal then it is not possible to delete the first character
I'll complete this post as I go along...
Memory size of X terminal emulators
Ideally I'd like to make do with the smallest applications which do exactly what I need and nothing more. In particular lets consider the memory footprint of terminal emulators i.e. xterm, mrxvt and my modified mrxvt.
I'll be using a non-debug build (no -g flag), "ls -ash" for the size of the executable and "ldd" to list the dependencies on shared libraries. I'll also be compiling them with Unicode support when possible.
xterm
316K xterm
needs 19 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libXaw.so.8 => /usr/X11R6/lib/libXaw.so.8 (0x401f1000)
libXmu.so.6 => /usr/X11R6/lib/libXmu.so.6 (0x4024d000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x40264000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x40272000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x402c5000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x402cd000)
libncurses.so.5 => /lib/libncurses.so.5 (0x402e5000)
libc.so.6 => /lib/tls/libc.so.6 (0x4032a000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x40444000)
libdl.so.2 => /lib/libdl.so.2 (0x40463000)
libXpm.so.4 => /usr/X11R6/lib/libXpm.so.4 (0x40467000)
libXp.so.6 => /usr/X11R6/lib/libXp.so.6 (0x40477000)
/lib/ld-linux.so.2 (0x40000000)
mrxvt with most options enabled
428K mrxvt
needs 18 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libpng.so.3 => /usr/lib/libpng.so.3 (0x401f1000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x4021f000)
libXpm.so.4 => /usr/X11R6/lib/libXpm.so.4 (0x4023f000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x4024f000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x40267000)
libz.so.1 => /lib/libz.so.1 (0x4026f000)
libm.so.6 => /lib/tls/libm.so.6 (0x40280000)
libc.so.6 => /lib/tls/libc.so.6 (0x402a3000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x403bd000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x403cb000)
libdl.so.2 => /lib/libdl.so.2 (0x403ea000)
/lib/ld-linux.so.2 (0x40000000)
My modified mrxvt
324K mrxvt
needs 15 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x401f1000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x40209000)
libz.so.1 => /lib/libz.so.1 (0x40212000)
libm.so.6 => /lib/tls/libm.so.6 (0x40223000)
libc.so.6 => /lib/tls/libc.so.6 (0x40246000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x4035f000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x4036d000)
libdl.so.2 => /lib/libdl.so.2 (0x4038c000)
/lib/ld-linux.so.2 (0x40000000)
By looking at my dependencies you may realise that I've removed a lot of visual features such as background png, jpeg support and transparency. I've also removed other stuff like some of the buttons that I deemed unnecessary.
Although the size of the executable plus the number of dependencies gives some idea at how much memory each application at runtime it's hard to put an exact figure as the programs don't run isolated.
Anyway my mission will be to once again make sure I only use the shared libraries I really need.
I'll be using a non-debug build (no -g flag), "ls -ash" for the size of the executable and "ldd" to list the dependencies on shared libraries. I'll also be compiling them with Unicode support when possible.
xterm
316K xterm
needs 19 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libXaw.so.8 => /usr/X11R6/lib/libXaw.so.8 (0x401f1000)
libXmu.so.6 => /usr/X11R6/lib/libXmu.so.6 (0x4024d000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x40264000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x40272000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x402c5000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x402cd000)
libncurses.so.5 => /lib/libncurses.so.5 (0x402e5000)
libc.so.6 => /lib/tls/libc.so.6 (0x4032a000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x40444000)
libdl.so.2 => /lib/libdl.so.2 (0x40463000)
libXpm.so.4 => /usr/X11R6/lib/libXpm.so.4 (0x40467000)
libXp.so.6 => /usr/X11R6/lib/libXp.so.6 (0x40477000)
/lib/ld-linux.so.2 (0x40000000)
mrxvt with most options enabled
428K mrxvt
needs 18 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libpng.so.3 => /usr/lib/libpng.so.3 (0x401f1000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x4021f000)
libXpm.so.4 => /usr/X11R6/lib/libXpm.so.4 (0x4023f000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x4024f000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x40267000)
libz.so.1 => /lib/libz.so.1 (0x4026f000)
libm.so.6 => /lib/tls/libm.so.6 (0x40280000)
libc.so.6 => /lib/tls/libc.so.6 (0x402a3000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x403bd000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x403cb000)
libdl.so.2 => /lib/libdl.so.2 (0x403ea000)
/lib/ld-linux.so.2 (0x40000000)
My modified mrxvt
324K mrxvt
needs 15 shared libraries
linux-gate.so.1 => (0xffffe000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x40019000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x4002c000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x4014c000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x401bb000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x401c3000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x401f1000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x40209000)
libz.so.1 => /lib/libz.so.1 (0x40212000)
libm.so.6 => /lib/tls/libm.so.6 (0x40223000)
libc.so.6 => /lib/tls/libc.so.6 (0x40246000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x4035f000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x4036d000)
libdl.so.2 => /lib/libdl.so.2 (0x4038c000)
/lib/ld-linux.so.2 (0x40000000)
By looking at my dependencies you may realise that I've removed a lot of visual features such as background png, jpeg support and transparency. I've also removed other stuff like some of the buttons that I deemed unnecessary.
Although the size of the executable plus the number of dependencies gives some idea at how much memory each application at runtime it's hard to put an exact figure as the programs don't run isolated.
Anyway my mission will be to once again make sure I only use the shared libraries I really need.
Tuesday 26 June 2007
Fvwm2rc 0.0
I've been working some more on my fvwm2rc file today.
As I have already mentionned what I consider my ideal desktop is:
1)one which is more or less completely keyboard-driven and where the key combinations are as efficient as possible.
2)a minimalistic one - all I need at the smallest cost possible and nothing more.
My current Fvwm consists of one desktop spread over 4 pages. To navigate efficiently across these 4 pages I have defined a few intuitive keyboard combinations. You can either use the MS Windows meta key along with the arrows or the h,j,k,l keys (vim style), or you can use the keypad to access any page in one key press (the keypad is a sort of map of the entire desktop instead of used for movement). I also have shortcuts for the 3 X applications I need the most: mrxvt (316K), firefox (9.5M) and thunderbird (12M). Most of the other appps use curses so run in mrxvt. As you can see thunderbird and firefox are quite large and require many libraries. I am planning on eventually superseding thunderbird with fetchmail+mutt. However because of the state of the Web and the current trend of Web 2.0, firefox will be harder to replace. I do however plan to experiment with simpler browser which I will configure and recompile for fun (lynx and links seem most accessible) but I don't see myself completely breaking away from firefox.
Back to my Fvwm...
At the top left corner lies an FvwmButtons panel with a pager and a digital xclock. The panel can be toggled to the foreground by the WindowsKey+Escape combination. This simple panel will eventually grow into a status panel where I will have my own choice of performance monitors, email notifiers and maybe other useful things.
Something one might consider peculiar with my setup is that I have partially done away with iconification. As a matter of fact, I never find the need for iconifying anything as I never run that many apps concurrently so have plenty of space with 4 pages (mrxvt and firefox and tabbed apps which each take up one page and thunderbird takes another, giving me one full page for anything else).
Anyway without further delay here's my current fvwm2rc file (this is not meant to be used as is but merely for reference and maybe to suscitate some feedback). Btw if any of you are also in the process of configuring you Fvwm may I suggest you take a look at FvwmConsole which allows you to configure Fvwm interactively instead restarting the window manager each time. Also here are 2 good examples of fvwm2rc files: http://linuxmafia.com/~n6tadam/fvwm/ and http://dev.gentoo.org/~taviso/fvwm2rc.html
Ok I've spent enough time on configuring my Fvwm for now. Tomorrow I'll commit my day to working on mrxvt.
As I have already mentionned what I consider my ideal desktop is:
1)one which is more or less completely keyboard-driven and where the key combinations are as efficient as possible.
2)a minimalistic one - all I need at the smallest cost possible and nothing more.
My current Fvwm consists of one desktop spread over 4 pages. To navigate efficiently across these 4 pages I have defined a few intuitive keyboard combinations. You can either use the MS Windows meta key along with the arrows or the h,j,k,l keys (vim style), or you can use the keypad to access any page in one key press (the keypad is a sort of map of the entire desktop instead of used for movement). I also have shortcuts for the 3 X applications I need the most: mrxvt (316K), firefox (9.5M) and thunderbird (12M). Most of the other appps use curses so run in mrxvt. As you can see thunderbird and firefox are quite large and require many libraries. I am planning on eventually superseding thunderbird with fetchmail+mutt. However because of the state of the Web and the current trend of Web 2.0, firefox will be harder to replace. I do however plan to experiment with simpler browser which I will configure and recompile for fun (lynx and links seem most accessible) but I don't see myself completely breaking away from firefox.
Back to my Fvwm...
At the top left corner lies an FvwmButtons panel with a pager and a digital xclock. The panel can be toggled to the foreground by the WindowsKey+Escape combination. This simple panel will eventually grow into a status panel where I will have my own choice of performance monitors, email notifiers and maybe other useful things.
Something one might consider peculiar with my setup is that I have partially done away with iconification. As a matter of fact, I never find the need for iconifying anything as I never run that many apps concurrently so have plenty of space with 4 pages (mrxvt and firefox and tabbed apps which each take up one page and thunderbird takes another, giving me one full page for anything else).
Anyway without further delay here's my current fvwm2rc file (this is not meant to be used as is but merely for reference and maybe to suscitate some feedback). Btw if any of you are also in the process of configuring you Fvwm may I suggest you take a look at FvwmConsole which allows you to configure Fvwm interactively instead restarting the window manager each time. Also here are 2 good examples of fvwm2rc files: http://linuxmafia.com/~n6tadam/fvwm/ and http://dev.gentoo.org/~taviso/fvwm2rc.html
Ok I've spent enough time on configuring my Fvwm for now. Tomorrow I'll commit my day to working on mrxvt.
Monday 25 June 2007
User applications
Overall
Minimalist and efficient. Ideally only run software which I can tweak for my own usage. Understand as much as possible. Be a control and performance freak.
The window manager: fvwm2
Small memory footprint. Keyboard driven. At most 2 applications per page at a time.
The terminal emulator: mrxvt
I am currently studying and modifying the code for my own use. I really like the tab support. Still not as complete as xterm (no UTF8 support for example). My mrxvt has been trimmed from all the visual stuff I deem unnecessary. I removed the -g flag from the final build and it is presently at 316K. I'm using the XFT functions for drawing fonts which don't support UTF-8 as of yet. I am hence currently looking at XFT functions which can either handle UTF-8 straight or maybe I'll do the conversion to another encoding first like UTF-16. More on this in next post...
The editor: vim
I am very impressed with vim but not that proficient yet. Smaller and faster than emacs. Very extensible for programming (ctag support).
The web browser: firefox or links?
This might be the most dependencies-hungry application I'll require. links is small and I have some hope of understanding and customising the source. firefox is huge and the code base seems huge. I don't think I'll be able to do with links only.
The email client: mutt?
I already use an ssh tunnel to get my mail from my POP server with thunderbird. Using fetchmail with mutt offer much better performance.
C code browsing tools
Exuberant ctags. Cscope. Cflow.
Minimalist and efficient. Ideally only run software which I can tweak for my own usage. Understand as much as possible. Be a control and performance freak.
The window manager: fvwm2
Small memory footprint. Keyboard driven. At most 2 applications per page at a time.
The terminal emulator: mrxvt
I am currently studying and modifying the code for my own use. I really like the tab support. Still not as complete as xterm (no UTF8 support for example). My mrxvt has been trimmed from all the visual stuff I deem unnecessary. I removed the -g flag from the final build and it is presently at 316K. I'm using the XFT functions for drawing fonts which don't support UTF-8 as of yet. I am hence currently looking at XFT functions which can either handle UTF-8 straight or maybe I'll do the conversion to another encoding first like UTF-16. More on this in next post...
The editor: vim
I am very impressed with vim but not that proficient yet. Smaller and faster than emacs. Very extensible for programming (ctag support).
The web browser: firefox or links?
This might be the most dependencies-hungry application I'll require. links is small and I have some hope of understanding and customising the source. firefox is huge and the code base seems huge. I don't think I'll be able to do with links only.
The email client: mutt?
I already use an ssh tunnel to get my mail from my POP server with thunderbird. Using fetchmail with mutt offer much better performance.
C code browsing tools
Exuberant ctags. Cscope. Cflow.
My ideal Linux - what I need and nothing more
I don't like using things I don't understand.
This was probably my rationale for studying Computer Science in the first place.
I should probably add: I don't like using things I don't understand or don't need.
I am not going to go into the psycho-analysis of why I and others think that way. All I am going to say is that this is both a good thing and bad thing. It is a bad thing because I get easily side-tracked (story of the man on the island that doesn't know what to do first). It is a good thing because it keeps me learning and creating. It also forces me to keep things simple.
Of course I still need to remain pragmatic as I don't have the brain or the time to figure everything out. This is why I start off with a system that does everything I need and more i.e. the Linux kernel with the required drivers, X11 and a few apps, then I customise by keeping or improving on the features I like and all features I do not require. Being somewhat of a spartan, this involves removing all the gloss and keeping a minimalistic, functional and efficient system.
My ultimate objective is to have a system completely tailored for my hardware and everyday usage - a Linux I can understand.
Minimalist and efficient setup. Ideally only run software which I need and can tweak for my own usage by editing the source.
Understand as much as possible in the process. Be a control and performance freak.
This was probably my rationale for studying Computer Science in the first place.
I should probably add: I don't like using things I don't understand or don't need.
I am not going to go into the psycho-analysis of why I and others think that way. All I am going to say is that this is both a good thing and bad thing. It is a bad thing because I get easily side-tracked (story of the man on the island that doesn't know what to do first). It is a good thing because it keeps me learning and creating. It also forces me to keep things simple.
Of course I still need to remain pragmatic as I don't have the brain or the time to figure everything out. This is why I start off with a system that does everything I need and more i.e. the Linux kernel with the required drivers, X11 and a few apps, then I customise by keeping or improving on the features I like and all features I do not require. Being somewhat of a spartan, this involves removing all the gloss and keeping a minimalistic, functional and efficient system.
My ultimate objective is to have a system completely tailored for my hardware and everyday usage - a Linux I can understand.
Subscribe to:
Posts (Atom)