Book Review: Python 2.6 Text Processing

Python is a powerful and dynamic programming language that is used in a wide variety of application domains such as web and internet development, databases access, desktop GUIs, scientific and numeric, education, network programming, software development, as well as games and 3D graphics.
As a security analyst I'm always interested in ways to better query vast quantities of text such as parsing web server logs for various signs of evil.
Jeff McNeil's Python 2.6 Text Processing Beginner's Guide from Packt Publishing struck me as useful resource with which to improve Python skills specific to text processing.
This book is intended for novice Python developers interested in processing text (me), and is laid out and written so as to be very supportive of this cause.
First published in December 2010, Python 2.6 Text Processing is organized via these conventions:

• Time for action - inclusive of multiple instructions followed by extra detail and explanation (What just happened?)
• Pop quiz - to help you test your understanding of methods just discussed
• Have a go hero - practical challenges to put your learning to use

I appreciate the logical flow of the book, moving from basic concepts and IO handling, to strings services and standard library usage, to regular expressions, structure markup, encoding, and advanced output.
With my interest in web server log manipulation I found myself able to quickly embrace the concepts offered and make us of this book's code samples offered on the Packt Publishing website.
Anyone who operates a website and spends any time reviewing web logs is likely aware that a certain percentage of all traffic bound for their site is malicious, be it uniquely targeted or bot traffic crawling by looking for weak spots.
One such example is remote file include (RFI) attempts. I've been using a Perl script to parse my logs for such traffic but have wanted to use such analysis as an opportunity to learn Python and ultimately rewriting the scripts in Python. While I haven't gotten there yet, I am certain this book will aid me entirely.
Of additional use is the fact that Python 2.6 Text Processing offers additional resources such documentation APIs, community resources such as mailing lists and conferences, as well as discussion of Python 3 and what to expect in migrating.
Returning to the RFI analysis mentioned above, I used Python to pull interesting, related results out of my web logs.
While Chapter 2 of Python 2.6 Text Processing introduces a web server log parser, and builds on it through out the chapter, I was drawn to searching and indexing as described in Chapter 11 via the use of the Nucular libraries (no, not the Bush mispronunciation).
"Nucular is a system for creating full text indices for fielded data. It can be accessed via a Python API or via a suite of command line interfaces."
First, ensure that you've installed the SetupTools easy_install system via python ez_setup.py as discussed on page 23. Once installed issue easy_install nucular, and the libraries and related dependencies will be installed to the appropriate paths.
With some modifications to the provided code samples, I then created an index of three years worth of web logs from my site, and was able to query them as a single source for keywords indicative of RFI attacks. While I started with a simple linear search across multiple logs via text_scan.py as seen on page 302 I quickly learned why McNeil is proving the linear search method as laborious and ineffective, instead promoting the use of libraries such as Nucular, and he's right.
Overall, this book is an effective learning tool, though keep in mind that it's entirely Linux-centric. Syntax for those of you using Python on Windows is subject to nuances.
McNeil's done a solid job with Python 2.6 Text Processing Beginner's Guide; it's a verbose (sometimes he turns on the fire hose) but worthy read and a suggested purchase at $45 +/- direct from Packt, Amazon, or Barnes and Noble, earning 3.5 stars out of 5 (very good).
Give it a read and put those mad new Python skills to good use.
Cheers.

del.icio.us | digg | Submit to Slashdot

Please support the Open Security Foundation (OSVDB)

Disclosure standards and why they're critical

If you've read this blog over the last couple of years you've likely made note of the varying degrees of success I've had disclosing vulnerabilities.
You've seen the best of breed in AppRiver and SmarterTools.
You've also seen lessons in how to not handle disclosures in the likes of American Express and Ameriprise. I believe Ameriprise is Pwnie-worthy for Lamest Vendor Response given that Benjamin Pratt, Ameriprise’s vice president of public communications, said "There's no one at risk here." and that there are no plans to review any of the mechanisms the company may have in place to receive notifications from the public about website vulnerabilities. Wow. The Consumerist clarified those statements aptly with "we assume he means, "No one important on our side of things. Our customers can suck it."

I take disclosure very seriously. I believe it is a deep seated, inherent responsibility that rests squarely on the shoulders of vendors and site operators. Equally, disclosure must be responsible, even when efforts to advise the vendor have come up empty. To that end: ReportSecurityFlaws.com, the result of a recent interview I gave to by Ira Victor of the Data Security Podcast on the topic of mishandled disclosures. We decided on a joint project, thus ReportSecurityFlaws.com.

Report Security Flaws exists to increase awareness and responsiveness in Internet vendors and web site operators when they receive security-related disclosures.
It is our hope that all vendors/operators maintain an email alias that exists for the sole purpose of receiving disclosure notices from parties reporting noted security flaws on the vendor/operator’s web site.

Further, said email alias should be monitored by individuals with an understanding of web application security issues and business logic flaws, while maintaining a close working relationship with the site developers and operations engineers. This relationship should allow for the quick escalation of reported issues for mitigation and remediation.
Examples of such email alias might include:
security@domain.com
websecurity@domain.com
webreports@domain.com

Too often vendors and web site operators fail to manage the proper intake and escalation of reported security flaws, leading to lapses in web application security for days, weeks, and even months.

Report Security Flaws will provide resources and guidance for vendors and site operators facing such challenges, with the hope of improving internet security posture for vendor/operators and consumers alike.

If you, dear reader, have tried to no avail to drive a site operator/web vendor/cloud provider to fix flaws, and received no reply, let us know. ReportSecurityFlaws intends to serve as a public motivator to close such gaps and promote improved vendor response, complete with standards.

NOTE: ReportSecurityFlaws is not intended to out vendors who fail to fix. Rather, it is to use all means necessary to ensure they do fix, and promote better standards and practices.

With standards in mind, I've been participating in discussions regarding ISO/IEC 29147, which will hopefully be embraced globally as the ISO standard Security
techniques – Responsible vulnerability disclosure.
We also believe there is an opportunity for the PCI Council to incorporate stringent disclosure practices in the PCI DSS.

Disclosure can and must be handled properly. This week's reading included a remarkable and detailed public incident report from Apache to the compromise they'd suffered the week prior. This kind of transparent, open response does, as they clearly state, make the internet a better place. Well done, Apache.

Stay tuned for more....
Cheers.

del.icio.us | digg | Submit to Slashdot

Please support the Open Security Foundation (OSVDB)