Categories
PHP Quick Post

AWK one-liners and Vim

Just a quick one – I had to edit an old PHP script that I wrote a while back to create calendar events. At the start of this file is a large array containing a number of letters which correspond to specific durations. As usual – I stuck with my one of the well known XKCD comics and decided that to update this file it was easier to learn a few new tricks (teach a man to fish and all that). In summary, I had to add to the file a list of formatted letters like

"N", 
"N",
"N",
"L",
"L"

Each letter was repeated 3 times on recurring lines, and there was a total of 189 letters which had to be wrapped in quotation marks and have a comma at the end.

To begin with, I created a text file and entered in each line one character that would eventually become 3 lines. I then wrapped this up in AWK using the following one liner;

awk '{ print "\"" $0 "\","; print "\"" $0 "\","; print "\"" $0 "\","}' input.txt > output.txt 

I know that docent look pretty, however its a one liner and it works. To start with,

print $0

will print the current line, so by prepending and appending the characters I needed, properly escaped, around that

$0

and repeating that 3 times I turned

N
L
R

into

"N",
"N",
"N",
"L",
"L",
"L",
"R",
"R",
"R",

for a total of 189 lines. I then moved this file to where I run my PHP script, and started VIM opening the two files at once. A quick press of

v 

to select Visual Mode where I select the whole file, followed by a

:b 1

to move to the PHP script (the b stands for buffer, and the 1 is because that was the first file opened. 2 would be the seconds etc.) in VIM and a quick paste and I was almost done – I just had to remove the final comma

,

from the last array entry. I’m not good enough to have a bit of AWK work the last bit out – and the time spent implementing it would not be worth it.

AWK one liners, and AWK itself has never really been something I’ve used. In the past its been presented to me as a way to solve problems, but the solutions have been handed to me. This is the first time I’ve actively went hunting for a solution without asking for help – and even though its simple, I’m quite happy I managed to figure it out – even if it did take about 20 minutes of searching…

As a final note I’m also quite happy I managed this completely in the command line. No mice were used in the editing of this script!

Categories
Hosting PHP

What to expect when your expecting… to be hacked

A bit of a disappointing one this. Theres nothing worse (well, I’m sure there is but in this context lets leave it as is) than receiving that dreaded e-mail from your host. It starts with a subject line similar to

Security Incident Concerning

and a body of text along the lines of

A security risk has been detected on your server.
We have been informed that your server contains or redirects to harmful or malicious content, such as malware or phishing sites.

Not at all ideal. In summary — the server in question began to host malicious content. It was from a domain that I don’t use and have a holding page only up, and a user at some point has reported the URL as a malware/phishing attempt. This then gets reported to my host, who then reports it to me.

After getting the email I was a bit perplexed as to how this site had been flagged as a security risk. I checked the URL given and sure enough, a redirect was in place taking it away from my server to some other (compromised) server. I thought it might have been a coding issue that allowed my domain to freely redirect pages (meaning any attacker could mask their own server with mine). I logged in and checked a few things. It wasn’t my code that was doing anything. I checked and seen a few other files that shouldn’t have been there, all with recent creation dates. A quick

find . -maxdepth 20 -mtime -20

netted me a few files that had been created 2 days prior. These were in a variety of directories, and as I spread my domains across a couple of servers these files also appeared in those directories. The suspect files all were all Base 64 encoded, and executed php scripts – given that they all started with

<?php eval(base64_decode('BASE64ENCODEDSTRINGHERE')) ?>

These files either redirected pages, contained a mass emailer (LeafPHPMailer) or opened up a (pretty feature rich but visually poor) file manager. I’m not going to go into too much detail but the reason for the infection came from one WordPress installation that I had completely forgot about after transferring to the new host. Its a site thats very seldom accessed, and to be honest, doesn’t require a WordPress Installation, but it was an easy CMS solution for someone.

Using an out-of-date plugin the attacker managed to place obfuscated PHP file on the server. This file was then accessed via a web browser which ran the PHP code, and allowed other files to be placed in different locations on the server.

When I deployed these servers I began hardening them against attacks like this. Unfortunately, I didn’t finish it. Some of my actions stopped the potential full-scale destruction of the server which I’m thankful for, but I’m a bit annoyed I didn’t finish my hardening steps.

Having separate users for different tasks on the server helped. This meant that any file modifications were only able to be done at the root web-directory level. Config files, where appropriate were hosted out-with the directory, and permissions meant that other files could not be modified. There were a few other steps that I’m not going to go into detail about however cashing up on a couple of guides on how to harden or secure your server should help.

After figuring out what happened, and how it happened, I stopped any public access – essentially shutting down the HTTP Daemon, removed any newly created files matching the time scales above, pretty deleted any WordPress installations and re-downloaded fresh copies of the MD5-checked files from wordpress.org, then manually checked all the database tables for Indicators of Compromise (IoC’s) line by line, entry by entry, and slowly reloaded everything.

I then finished my hardening that I should have done before.

I think it’s important to vent that “hacking” isn’t hacking any more. It’s what used to be known as “Script Kiddies” who are now essentially Serious and Organised Crime Groups that use these phishing and malware scams to extract money. Theres no hacking in the traditional sense – just the unauthorised access to computers that wreck havoc on people who are caught by it. It’s the same as a teenager using a Low Orbit Ion Cannon.

Am I embarrassed this happened to me? Of course. Annoyed? Yep. But I’m also relieved that it did — it means I was able to stop it before it became much, much worse.

Categories
Electronics PHP

Just a little bit more.

That’s what I’ve been like over the last few days. Every time I try to move away from my computer I think of something I can change, or a small bug that I can fix over at whatplanshouldiget.uk.

I mentioned that site in my last post. It was going to be a quick script. A quick play around with PHP to keep me fresh, however still help me decide what price plan to get the new iPhone on, and possibly help others. I think, today, barring any bugs being found, it’s finished – excluding extra price plans being entered from MNVO’s.

A total of 1259 lines of my own code spread over 5 files. That’s not a lot in the grand scheme of things. That’s not a lot point-blank. But for something that started out as a simple exercise  turned into a little bit more. I’m proud of it. My next task, being as its possibly only valid for a few months until the UK Carriers start offering cut price plans, is to turn it from procedural code into object orientated code.

I’ve been using PHP since Version 4. Version 4 was really easy to get into. It wasn’t strict with coding styles (as long as you followed the syntax), and has garnered a reputation as helping to create the worst developers in the world. PHP 4 was almost exclusively procedural, as well as a whole host of other things I won’t even pretend to understand. PHP 5 came along and changed all that, and I didn’t really keep up with the times. I’m aware of some of the syntax, and the reasons, but I still don’t fully grasp the idea of OOP. This small site will be a great stepping stone!

Categories
Electronics General PHP

A new iPhone?

Well. It’s sort of guaranteed that I’ll be buying the new iPhone when its launched. The announcement is in 4 days, and the Keynote will be streamed live. I hope it’s streamed via the Apple TV again.

Countdown to Apple Event
Countdown to Apple Event

I’m not going to post about the handset or the specifications. That has been done again and again.

My biggest issue when it’s time for me to upgrade my handset is trying to figure out the best plan, and whether its worth purchasing the handset outright and taking a SIM only deal, or taking out an actual contract.

In the UK, we have a number of options for phones. The first (and most common) option is to take out a (now standard) 24 month contract. You pay more per month, but in that monthly payment you pay for the handset and for the service. Taking an iPhone 5S for example, I can grab a 16Gb iPhone 5S on the o2 network with Unlimited Minutes and Texts with 2Gb of 4G data for £43 per month, with no charge for the handset. Thats it. You walk in, get a phone, no payment, and walk out. Alternatively, I can pay £49.99 for the handset with the above options, and pay £38 per month. So, two options already and thats for the same contract. Doing the math and basing it on the average monthly cost, ( (£38 x 24) + £49.99 = £40.08 vs £42) shows the better deal to be £40.08.

Then we have SIM only deals. This is where you purchase the device out right (usually at Apple prices, then its “Factory Unlocked” – gives a greater re-sale value) and then pay monthly just for the plan. Again, you get different deals if you take out a “rolling” 30 day contract, or a 12 month contract. Again, using o2 as an example, the same plan as above would cost £22 per month for a 30 day rolling contract, vs £20 per month for a 12 month contract. Again, doing the math and basing it on the average monthly cost over 2 years, factoring in the price of an iPhone 5S 16Gb ( (£20 x 24) + £549.00 = £42.87) shows the better deal to be £40.08, i.e paying £49.99 for the handset and paying £38 per month. I’ve only included the £20 per month tariff as I’m basing this on a 24 month contract.

So, I have maths. Thats only comparing 1 plan with 3 options. To do this with every tariff available in the UK with the various handsets would take a fair bit of time. To that end, I created a website. WhatPlanShouldIGet.uk. Its pretty simple. Right now, it only allows you to enter a purchase price for a handset. As soon as the new mobile tariffs are announced for the iPhone 6 (or whatever it will be called), I will update the site with the various bits of data required. You will then be able to choose the handset, and the site will give you a nice, already worked-out table for your comparing pleasure!

The above assumes a lot of things. It assumes you have the cash available to purchase an iPhone outright. It does not take into account any interest rates imposed by credit card companies if you purchase the handset on a credit card (although that is a pretty good idea – new feature request!) and it does not take into account any loyalty or other discounts you may be offered.

It’s just a simple, quick and easy tool to work out What Plan Should I Get.

The site itself is written in PHP. It uses MySQL for backend storage, and bootstrap to make it look nice. I’ll post the code at some point, but as its a public facing site, I want to make sure I haven’t left any data destroying bugs (either server or client side) in the code. For more information thought, you can head over to WhatPlanShouldIGet.uk

Categories
Electronics notes PHP

OSX 10.8.2 localhost directory & MySql / PHPMyAdmin

You know, just because it will bug me again at some point:

If for any reason you don’t want to use the “sites” folder for your web development, you can head to

/Library/WebServer/Documents

and upload code directly into your http://localhost folder.

to install PHPMyAdmin, just plop that PHPMyAdmin folder just downloaded right into the above folder.

To Install MySQL however, follow these steps:

Download the .dmg MySQL installed from MySQL.com

Run the first installed, then the auto-start installer, and finally the Preference Pane add-on.

Run this code in terminal:

sudo cp /etc/php.ini.default /etc/php.ini

and then load /etc/php.ini and replace any instance of

/var/mysql/mysql.sock

with

/tmp/mysql.sock

Start the MySQL server using the preference pane add-on, then run the following in Terminal:

/usr/local/mysql/bin/mysqladmin -u root password PASSWORD

Replacing PASSWORD with a password of your choice.

The last thing to do is head to the PHPMyAdmin directory, and copy the config.sample.inc.php to config.inc.php and edit that file, replacing the

$cfg['blowfish_secret'] = 'RANDOM BLOWFISH CHARACTERS'; /* YOU MUST FILL IN THIS FOR COOKIE AUTH! */

RANDOM BLOWFISH CHARACTERS part with random characters!

Finally, head to http://localhost/phpmyadmin and your good to go!

Categories
Electronics General GitHub PHP

Using only PHP to save Google Starred items to Pocket!

So, based on my last post, I wanted to see if I could do everything with PHP. After a bit of google-fu and using the php.net manual, I’ve managed this beauty. Use at your own risk! This works with my Google Starred items, and you still have to obtain the starred.json file from the Google Take Out service (See my last post for more information).

If you have any tips on how to improve this, drop me a comment!

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Instapaper: Export</title>
</head>
<body>
<h1>Unread</h1>
<ol>
<?php

// First off, we start by opening the file required (starred.json),
// Then we set the $galbool paramater (This is used where sites have
// given a "Gallery" URL (To make it more cosmetic, it appends the
// text [Gallery] to the end of the description.

$file_handle = fopen("starred.json", "r");
while (!feof($file_handle)) {
    $galbool = FALSE;
    $line = fgets($file_handle);

// This is our first check. We run through the json file and look for
// lines that contain the text "href". If it does not have that text,
// we are not interested, and set that line to be blank.

    $preg = "(\"href\")";
    $urlcheck = preg_match($preg, $line);
    if ($urlcheck !== 1) {
    $line = "";
    } else {

// A cheeky little bit of coding. Whislt we are in the loop, and
// I know that this is a URL we are intersted in, I'll have a look
// at the last character. If its a ",", I also want to delete that
// line. Looking at the JSON file, if a line contains a URL and
// ends with a ",", it means its not the *ACTUAL* URL we want, so
// we continue our ruthless streak and set that line to blank!
// (This was included ot deal with hackaday.com URL's, which for
// some reason doubled up, and this was a quick and easy way to
// get rid of them!

        $preg = "(,)";
        $clean = preg_match($preg, $line);
        if ($clean !== 0) {
            $line = "";
        }
        }

// Now we trim the whitespace and other non-needed characters, and
// we remove the first bit from the string thats not needed. This
// takes us right to the http:// part of the link, which is what we
// need! We also remove the trailing slash from the link as well.

        trim($line);
        $line = substr($line, 16);
        $line = substr_replace($line, "", -2);
        $string = $line;

        $check = $string[strlen($string)-2];
        if ( $check == "/"){
            $string = rtrim($string);
            $string = rtrim($string, "/");
            $desc = $string;
        }
        $check = $string[strlen($string)-1];
        if ( $check == "/"){
            $string = rtrim($string);
            $string = rtrim($string, "/");
            $desc = $string;
        }

// This is just a quick check to see if the URL passed is a gallery
// URL. If so, we set the $galbool value to true, and then do our
// usual URL cleanup. I have removed the /gallery part from the URL
// This is personal preferance, and I've not had any adverse effects
// from either taking it in, or removing it. It has to be removed
// just now to make figuring out the link text easier though.
// We can add it back in later if required.

        $gallerycheck = str_replace("/gallery", "", $string, $count);
        if ($count == 1){
            $galbool = TRUE;
            $string = rtrim($string);
            $string = rtrim($string, "/gallery");
            $desc = $string;
        }

// And now for the (almost) finale! We take everything after the
// forward slash in the URL, remove that forward slash, then we
// run through and replace every "-" with a space. This makes the
// end HTML page look nice, and it keeps with Instapapers Export
// option. If the $galbool value is true, we create a [Gallery]
// tag.

        $desc = strrchr($string, "/");
        $desc = str_replace("/", "", $desc);
        $desc = str_replace("-", " ", $desc);
        $desc = ucwords($desc);
        if ($string != "" ){
        if ($galbool == TRUE){

// you can add back in the /gallery link here again if you need it!
// Just uncomment the relevant line and comment out the other!
//        -------------------------------------------------------------------------------------------------

//        $formatted = '            <li><a href="' . $string . '/gallery">' . $desc . '[Gallery]</a></li>';

//        ----------------------------------***OR THIS LINE***---------------------------------------------

        $formatted = '            <li><a href="' . $string . '">' . $desc . '[Gallery]</a></li>';

//        -------------------------------------------------------------------------------------------------

        } else {
        $formatted = '            <li><a href="' . $string . '">' . $desc . '</a></li>';
        }
        echo $formatted;
        }
    }

// We now close the file (good housekeeping), and finish up
// the script.

fclose($file_handle);
?>
</ol>
</body>
</html>

 

Categories
Electronics GitHub PHP

Using the terminal & PHP to save Google Reader Starred items!

*MAJOR UPDATE* – USING PHP ONLY WITH NEW CODE! Please view this next post for more information!

So. Google Reader is closing down. I’m not going to get all high and mighty – It’s Google’s product. They do with it as they wish! There are several places that let you save your feeds from Google Reader, but I wanted to add all my Starred Items from Reader into Pocket. It turns out it wasn’t the easiest thing to do! After a few terminal commands and some PHP, this is what I came up with!

Firstly, head to Google Reader, and use the Data Takeout feature that Google provides to save your Reader Data only. The outputted ZIP file should contain a folder entitled “Reader”. Within that, there should be a file named “starred.json”.

*UPDATE* – I have saved these 2 files to GitHub! Go, Grab!
https://github.com/nickwebcouk/pocketimport

Now the fun begins! To make it easy (and quick) I used the Terminal on Mac and ran these commands within the above folder. I used two files (new.txt and newnew.txt) just to keep track of what was happening. There are easier ways of doing this! I ran the following commands from terminal:

grep -a1 "canonical" starred.json > new.txt
grep -v "^\--" new.txt > newnew.txt
grep -v "} ]," newnew.txt > new.txt
grep -v "\"canonical\" : \[ {" new.txt > newnew.txt
grep -v "\"updated" newnew.txt > new.txt
grep -v "} \]," new.txt > newnew.txt
cat newnew.txt | rev | cut -c 2- | rev > new.txt
cut -c 17- new.txt > newnew.txt
rm -r new.txt
mv newnew.txt url.txt

The “grep” command looks through text files for specific expressions. “cat” outputs a full file, “rev” reverses items, “cut” cuts text, “rm” removes files and “mv” moves files.

To make this easier, I created a Shell Script (Tested on OSX 10.8.2 only)

#!/bin/sh

clear
grep -a1 "canonical" starred.json > new.txt
grep -v "^\--" new.txt > newnew.txt
grep -v "} ]," newnew.txt > new.txt
grep -v "\"canonical\" : \[ {" new.txt > newnew.txt
grep -v "\"updated" newnew.txt > new.txt
grep -v "} \]," new.txt > newnew.txt
cat newnew.txt | rev | cut -c 2- | rev > new.txt
cut -c 17- new.txt > newnew.txt
rm -r new.txt
mv newnew.txt url.txt

You can run that file by saving it to the same location as the Google Reader folder, and running the following in terminal first:

chmod 755 script.sh
./script.sh

the first line tells the computer to allow script.sh to be executed, and the second line executes the script.

I then moved the url.txt file that had just been created to my www root folder (for me its under /users/~name/sites/), and created the following PHP/HTML code:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Instapaper: Export</title>
</head>
<body>
<h1>Unread</h1>
<ol>
<?php
    $file_handle = fopen("url.txt", "r");
    while (!feof($file_handle)) {
        $galbool = FALSE;
        $line = fgets($file_handle);
        $string = $line;
        $check = $string[strlen($string)-2];
        if ( $check == "/"){
            $string = rtrim($string);
            $string = rtrim($string, "/");
            $desc = $string;
        }
        $gallerycheck = str_replace("/gallery", "", $string, $count);
        if ($count == 1){
            $galbool = TRUE;
            $string = rtrim($string);
            $string = rtrim($string, "/gallery");
            $desc = $string;
        }
        $desc = strrchr($string, "/");
        $desc = str_replace("/", "", $desc);
        $desc = str_replace("-", " ", $desc);
        $desc = ucwords($desc);
        if ($galbool == TRUE){
        $formatted = '			<li><a href="' . $string . '">' . $desc . '[Gallery]</a></li>';
        } else {
        $formatted = '			<li><a href="' . $string . '">' . $desc . '</a></li>';
        }
        echo $formatted;
    }
    fclose($file_handle);
?>
</ol>
</body>
</html>

This provided me with a HTML page, which if saved as instapaper-export.html allowed me to head to getpocket.com and use the Instapaper import option.

584 starred articles and 2 seconds later, I received this wonderful little message!

getpocket import