Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Wednesday, July 15, 2015

Why testing is good and TDD evil

In our lives nowadays we have safety nets everywhere. We are scared and live in a world of scaremongering. Many people are even scared to go outside. They just sit in front of their computers afraid of failure. What if I'm not seen in the best light? What if my application fails? What if someone sees my code's not perfect?

But the problems are not failures, errors and epic fails - the problem is our inability or unwillingness to accept they're here to stay. It's about how we respond to failures not about how we make sure there are no failures. It's about us accepting failures are inevitable then spending time writing tests for every possible scenario so we have excuses.

Let's be brave - hope for the best and be prepared for the worst. Roll up our sleeves take a leap of faith and change that bit of "untested" code. If there are problems let's fix them. Have a roll back solution at hand so you could revert back if your code change breaks things up.
TDD is an epic failure. And rightly so. It instils false sense of security through passing tests and creates culture of excuses through so called "test coverage" and "code metrics" - and we all know how bosses like metrics, don't they.

Let me repeat again - TDD is not cure. You can find bugs in every software. TDD was meant to eradicate bugs but the bugs are still around. TDD actually makes it harder to tackle bugs because all your passing tests lead developers to believe that the bugs are actually caused by infrastructure, 3rd party code and so on. You can always come around and say "but all my tests are passing so it's not my problem".

And let's be clear here. Tests are code. The same code we try to tests. So why your code may be buggy and your tests are always right? And what does it mean 100% test coverage? Is the code really tested to work correctly (whatever that means) with all possible input parameters and changing condition? To give you an example have a look it this Java code:

public class Adder {
    int addOne(int input) {
        return input++;
    }
}

public class AdderTest {
    @Test
    public void addOneWorksWithNumber1() {
        int result = Adder.addOne(1);
        assertEquals(2, result);
    }

    @Test
    public void addOneWorksWithNumber0() {
        int result = Adder.addOne(0);
        assertEquals(1, result);
    }
}

Is this code thoroughly tested? It appears so. Test coverage is 100%, tests passing but what value the test gives you? To answer that lets add this test:

@Test
public void addOneWorksWithVeryLargeNumber() {
    int result = Adder.addOne(0xFFFFFFFF);
    assertEquals(0x100000000L, result);
}

It's not passing! And you could probably test other edge cases. In short, we tests ONE line of code, we've already written 30 lines code to test it but we're still not 100% sure the code does what we expect it to do. So the tests are just a smoke screen. The code can still fail, it's not tested for null values, for multithreading, for runtime exceptions and so on. You could spend years of writing tests and the code could be still buggy. Remember one thing - only the code you don't write can't fail. The less code you write, the better.

Other aspect of TDD is rigidness of the solution. Any code change, and changes are the only thing that doesn't change, usually results in either retiring some tests or worse amending them. So developers try to resist changes more. They know that they will have to not only change the code, but also God knows how many tests directly or indirectly related to the code they're about to change.

TDD also leads to the new blame culture. You commit and now tests are broken. And your name is dragged around like if you were worse than Usama Bin Laden. Some organizations go to that lengths as to punish (or promote) developers according to build failures. So it pays off to sit in the corner and touch nothing.

So instead of fighting this futile battle let's just accept we can't win here by simply planning our defenses. This is akin to playing a game of chess  by simply laying out the best possible defense scenario. No matter what, you can't win this way. You have to go out to the battle and prove you can win.

I mean don't try to avoid the battle. Do not try to defer the battle by writing super defensive code. It only slows you down, distracts you from the real target. Focus on writing maintainable code, focus on quick release cycles (ideally be able to only minutes between committing you bug fix and this bug fix being released to test and prod environment), focus on clean code, use defensive programming for mission critical modules and so on. Because what really matters in the end of the day isn't bug free code but your ability to quickly fix bugs for only one thing is sure in this world - your code is buggy!

To avoid any confusion - I'm not against TDD per se (and definitely not against testing in general). It makes sense to have tests making sure core parts of your functionality (usually something related to finance calculations) are covered by tests. It also makes sense to run automated sanity testing after your CI build.

Addendum: This has nothing to do with test frameworks. They are actually quite good when it comes to ad hoc testing and verifying some functionality both in isolation (unit) and as part of a bigger system (integration).

It also makes sense to have plentiful of tests if you're lucky enough to work on a framework, library or something which is intended to be highly re-useable.

Monday, November 21, 2011

Challenging week

What a week last week. Three project - three completely different technologies.
  • I started the week bug fixing our Wheelchair portal written in Grails. Grails underpinned by my favorite language Groovy is by far framework of my choice and means that i had a pretty good time coding.
  • Next one - PoC opening Oracle CRM (EBiz suite) custom form preloaded with data coming from our new out of the shelf leasing product. Normally we would pass this information in URL but that would mean customizing Oracle CRM which is expensive and no one wants to do it. Solution - Calling system will store input data in database and CRM will load this data on form load event. This event can be handled as personalization (not customization). Technologywise - Oracle SOA to store data into database (exposed as sync webservice) and Oracle forms on CRM side.
  • Last one - creating MS Outlook tasks triggered by events in CRM. The CRM bit is outsourced but the Exchange bit is pure java witha hint of Apache Jackrabbit (client side only - no webDAV server) and composition of webDAV messages.
To be honest that somehow demonstrate how experienced developers (could this be me?) can contribute - by bringing in knowledge from many fields spanning across many technologies. I'm not saying that less experienced developers can't do that but in my opinion they're more focus on latest trends rather than having all round knowledge especially outside of the mainstream (how many of them do Oracle forms, BPEL, not to mention webDAV?).
Anyway good times last week, this week it will be more about implementing more PoC and more, unfortunately, meetings ...

Thursday, November 17, 2011

Creating an Outlook task in Java using webDAV

I've recently come across this interesting task. In Java, create MS Exchange tasks with assigned reminders and custom task body. I was told that PoC (proof of concept) had been successfully concluded and it was just a matter of integration. To be honest, the PoC came up with this idea - create a web application which would utilize jIntegra DCOM wrapper (as the web would be running in Linux so no COM there) and remotely access MS Outlook installed somewhere and from there it would create tasks in Exchange. As you can imagine, the level of complexity of this approach is enormous. This solution would do a round trip around the world just to access an Exchange server and create a task there. Still this solution would eventually fail because of Outlooks scripting protection (see it here) and would require someone permanently logged in Outlook so DCOM calls would be possible.
What other (if any) solutions are out there. One pretty straight forward and surprisingly Java friendly too. Long time ago someone invented this extension to HTTP called webDAV. Yes, it's still around - alive and kicking. As webDAV is generally speaking just a collection of additional HTTP verbs (methods) it can be easily invoked from any language as long as the program can access HTTP (i.e. all).
Below's how I did it. You only need to download Jackrabbit jar ( freely available from Apache) and run the code. It internally performs a webDAV call that creates a new Exchange task. Tested with Exchange 2003 SP2 but should work with any 2003+ version. Additional files can be found under the main one. It's just a small tweak to Jackrabbit's own PropPatch method (adding PropPatch call with the XML in a String) and HtmlEncode helper (not my invention!) class to allow html task body encoding.

More info and source can be found at these URLs:
package uk.co.mo.webDAV.utils;

import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.params.HttpConnectionManagerParams;

import java.util.Date;


public class TaskCreator {

    public static void main(String[] args) throws Exception {

        // create a httpclient config
        String uri = "http://xxx.yyy.zzz/exchange/";
        // String uri = "http://xxx.yyy.zzz/exchange/";
        HostConfiguration hostConfig = new HostConfiguration();
        hostConfig.setHost(uri);

        HttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
        HttpConnectionManagerParams params = new HttpConnectionManagerParams();
        int maxHostConnections = 20;
        params.setMaxConnectionsPerHost(hostConfig, maxHostConnections);
        connectionManager.setParams(params);

        HttpClient client = new HttpClient(connectionManager);
        client.setHostConfiguration(hostConfig);

        // set authentication (most exchange servers accept nt challenge-response)
        Credentials creds = new NTCredentials("admin_username", "password", "ip", "domain");
        // Credentials creds = new NTCredentials("username", "password", "domain controller", "domain");
        client.getState().setCredentials(AuthScope.ANY, creds);

        PropPatchMethodPro pPatch = new uk.co.mo.webDAV.utils.PropPatchMethodPro("http://xxx.yyy.zzz/exchange/user/Tasks/nameOfThetask" + new Date().getTime() + ".eml", getMessage());
        // PropPatchMethodPro pPatch = new uk.co.mo.webDAV.utils.PropPatchMethodPro ("http://xxx.yyy.zzz/exchange/user/Tasks/nameOftheTask") new Date().getTime() + ".eml", getMessage());

        pPatch.setRequestHeader("Content-type", "text/xml; charset=UTF-8");

        // call exchange
        client.executeMethod(pPatch);

        // is everything OK
        if (!pPatch.isSuccess(pPatch.getStatusCode())) {
            // we've got a problem
            System.out.println("PROBLEM");
            System.out.println(pPatch.getStatusCode());
            System.out.println(pPatch.getStatusText());
            System.out.println(pPatch.getStatusLine());
        } else {
            // processed
            System.out.println("OK");
        }
    }

    private static String getMessage() {
        return new StringBuilder("<?xml version=\"1.0\"?>\n")
                .append("<g:propertyupdate xmlns:g=\"DAV:\"\n")
                .append("xmlns:e=\"http://schemas.microsoft.com/exchange/\"\n")
                .append("xmlns:mapi=\"http://schemas.microsoft.com/mapi/\"\n")
                .append("xmlns:mapit=\"http://schemas.microsoft.com/mapi/proptag/\"\n")
                .append("xmlns:dt=\"urn:uuid:c2f41010-65b3-11d1-a29f-00aa00c14882/\"\n")
                .append("xmlns:h=\"http://schemas.microsoft.com/mapi/id/{00062003-0000-0000-C000-000000000046}/\"\n")
                .append("xmlns:i=\"http://schemas.microsoft.com/mapi/id/{00062008-0000-0000-C000-000000000046}/\"\n")
                .append("xmlns:header=\"urn:schemas:mailheader:\"\n")
                .append("xmlns:sql=\"urn:schemas-microsoft-com:mapping-schema\"\n")
                .append("xmlns:mail=\"urn:schemas:httpmail:\"\n")
                .append("xmlns:t=\"http://schemas.microsoft.com/exchange/tasks/\">\n")
                .append("<g:set>\n")
                .append("<g:prop>\n")
                .append("<g:contentclass>urn:content-classes:task</g:contentclass>\n")
                .append("<e:outlookmessageclass>IPM.Task</e:outlookmessageclass>\n") // we're creating a task
                .append("<mail:subject>Test Task</mail:subject>\n") //subject line of the task
                .append("<mail:htmldescription>") // html message - needs to html encoded
                .append(HtmlEncode.text("Task Test<br/><b>test bold</b><pre>test pre</pre><a href='www.google.com'>www.google.com</a>"))
                .append("</mail:htmldescription>\n")
                .append("<h:0x8104 dt:dt=\"dateTime.tz\">2011-11-20T01:00:00.00Z</h:0x8104>\n") //start date
                .append("<h:0x8105 dt:dt=\"dateTime.tz\">2011-11-21T01:00:00.00Z</h:0x8105>\n") //due date
                .append("<i:0x8516 dt:dt=\"dateTime.tz\">2011-11-20T01:00:00.00Z</i:0x8516>\n") //start date
                .append("<i:0x8517 dt:dt=\"dateTime.tz\">2011-11-21T01:00:00.00Z</i:0x8517>\n") //due date
                .append("<i:0x8502 dt:dt=\"dateTime.tz\">2011-11-20T15:50:00.00Z</i:0x8502>\n") // reminder date
                .append("<h:0x811C dt:dt=\"boolean\">0</h:0x811C>\n")  // completed
                .append("<h:0x8101 dt:dt=\"int\">0</h:0x8101>\n") // status
                .append("<h:0x8102 dt:dt=\"float\">0</h:0x8102>\n") //completed %
                .append("<i:0x8503 dt:dt=\"boolean\">1</i:0x8503>\n")  // reminder enabled
                .append("<header:to>apollosystem</header:to>\n") // task destination
                .append("</g:prop>\n")
                .append("</g:set>\n")
                .append("</g:propertyupdate>").toString();
    }
}
package uk.co.mo.webDAV.utils;

import org.apache.commons.httpclient.methods.StringRequestEntity;
import org.apache.jackrabbit.webdav.DavConstants;
import org.apache.jackrabbit.webdav.DavMethods;
import org.apache.jackrabbit.webdav.client.methods.DavMethodBase;
import org.apache.jackrabbit.webdav.property.DavPropertyNameSet;
import org.apache.jackrabbit.webdav.property.DavPropertyName;
import org.apache.jackrabbit.webdav.property.DavPropertyNameIterator;
import org.apache.jackrabbit.webdav.DavServletResponse;
import org.apache.jackrabbit.webdav.MultiStatus;
import org.apache.jackrabbit.webdav.MultiStatusResponse;
import org.apache.jackrabbit.webdav.DavException;
import org.apache.jackrabbit.webdav.Status;
import org.apache.jackrabbit.webdav.xml.DomUtil;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.HttpConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

import java.io.IOException;

public class PropPatchMethodPro extends DavMethodBase implements DavConstants {
    private static Logger log = LoggerFactory.getLogger(PropPatchMethodPro.class);

    private final DavPropertyNameSet propertyNames = new DavPropertyNameSet();

    private DavException responseException;

    /**
     * @param uri
     * @throws IOException
     */
    public PropPatchMethodPro(String uri, Document document) throws IOException {
        super(uri);
        setRequestBody(document);
    }

    public PropPatchMethodPro(String uri, String document) throws IOException {
        super(uri);
        setRequestBodyText(document);
    }

    public void setRequestBodyText(String requestBody) {
        setRequestEntity(new StringRequestEntity(requestBody));
    }


    private Element getPropElement(Element propUpdate, boolean isSet) {
        Element updateEntry = DomUtil.addChildElement(propUpdate, (isSet) ? XML_SET : XML_REMOVE, NAMESPACE);
        return DomUtil.addChildElement(updateEntry, XML_PROP, NAMESPACE);
    }

    //---------------------------------------------------------< HttpMethod >---

    /**
     * @see org.apache.commons.httpclient.HttpMethod#getName()
     */
    @Override
    public String getName() {
        return DavMethods.METHOD_PROPPATCH;
    }

    //------------------------------------------------------< DavMethodBase >---

    /**
     * @param statusCode
     * @return true if status code is {@link DavServletResponse#SC_MULTI_STATUS 207 (Multi-Status)}.
     *         For compliance reason {@link DavServletResponse#SC_OK 200 (OK)} is
     *         interpreted as successful response as well.
     */
    @Override
    protected boolean isSuccess(int statusCode) {
        return statusCode == DavServletResponse.SC_MULTI_STATUS || statusCode == DavServletResponse.SC_OK;
    }

    /**
     * @param multiStatus
     * @param httpState
     * @param httpConnection
     */
    @Override
    protected void processMultiStatusBody(MultiStatus multiStatus, HttpState httpState, HttpConnection httpConnection) {
        // check of OK response contains all set/remove properties
        MultiStatusResponse[] resp = multiStatus.getResponses();
        if (resp.length != 1) {
            log.warn("Expected a single multi-status response in PROPPATCH.");
        }
        boolean success = true;
        // only check the first ms-response
        for (int i = 0; i < 1; i++) {
            DavPropertyNameSet okSet = resp[i].getPropertyNames(DavServletResponse.SC_OK);
            if (okSet.isEmpty()) {
                log.debug("PROPPATCH failed: No 'OK' response found for resource " + resp[i].getHref());
                success = false;
            } else {
                DavPropertyNameIterator it = propertyNames.iterator();
                while (it.hasNext()) {
                    DavPropertyName pn = it.nextPropertyName();
                    success = okSet.remove(pn);
                }
            }
            if (!okSet.isEmpty()) {
                StringBuffer b = new StringBuffer("The following properties outside of the original request where set or removed: ");
                DavPropertyNameIterator it = okSet.iterator();
                while (it.hasNext()) {
                    b.append(it.nextPropertyName().toString()).append("; ");
                }
                log.warn(b.toString());
            }
        }
        // if  build the error message
        if (!success) {
            Status[] st = resp[0].getStatus();
            // TODO: respect multiple error reasons (not only the first one)
            for (int i = 0; i < st.length && responseException == null; i++) {
                switch (st[i].getStatusCode()) {
                    case DavServletResponse.SC_FAILED_DEPENDENCY:
                        // ignore
                        break;
                    default:
                        responseException = new DavException(st[i].getStatusCode());
                }
            }
        }
    }

    /**
     * @return
     * @throws IOException
     */
    @Override
    public DavException getResponseException() throws IOException {
        checkUsed();
        if (getSuccess()) {
            String msg = "Cannot retrieve exception from successful response.";
            log.warn(msg);
            throw new IllegalStateException(msg);
        }
        if (responseException != null) {
            return responseException;
        } else {
            return super.getResponseException();
        }
    }
}
package uk.co.mo.webDAV.utils;

import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;

/**
 * Encodes text and URL strings in various ways resulting HTML-safe text.
 * All methods are <code>null</code> safe.
 */
public class HtmlEncode {

    protected static String EMPTY_STRING = "";

    protected static final char[][] TEXT = new char[64][];
    protected static final char[][] BLOCK = new char[64][];
    protected static final char[][] URL = new char[256][];

    /**
     * Creates HTML lookup tables for faster encoding.
     */
    static {
        for (int i = 0; i < 64; i++) {
            TEXT[i] = new char[]{(char) i};
        }
        for (char c = 0; c < 256; c++) {
            try {
                URL[c] = URLEncoder.encode(String.valueOf(c),
                        "ISO-8859-1").toCharArray();
            } catch (UnsupportedEncodingException ueex) {
                ueex.printStackTrace();
            }
        }

        // special HTML characters
        TEXT['\''] = "'".toCharArray(); // apostrophe (''' doesn't work - it is not by the w3 specs)
        TEXT['"'] = """.toCharArray(); // double quote
        TEXT['&'] = "&".toCharArray(); // ampersand
        TEXT['<'] = "<".toCharArray(); // lower than
        TEXT['>'] = ">".toCharArray(); // greater than

        // text table
        System.arraycopy(TEXT, 0, BLOCK, 0, 64);
        BLOCK['\n'] = "<br>".toCharArray(); // ascii 10, new line
        BLOCK['\r'] = "<br>".toCharArray(); // ascii 13, carriage return
    }

    // ---------------------------------------------------------------- encode

    public static String text(Object object) {
        if (object == null) {
            return EMPTY_STRING;
        }
        return text(object.toString());
    }

    /**
     * Encodes a string to HTML-safe text. The following characters are replaced:
     * <ul>
     * <li>' with &#039; (&apos; doesn't work)</li>
     * <li>" with &quot;</li>
     * <li>& with &amp;</li>
     * <li>< with &lt;</li>
     * <li>> with &gt;</li>
     * </ul>
     *
     * @see #block(String)
     */
    public static String text(String text) {
        int len;
        if ((text == null) || ((len = text.length()) == 0)) {
            return EMPTY_STRING;
        }
        StringBuilder buffer = new StringBuilder(len + (len >> 2));
        for (int i = 0; i < len; i++) {
            char c = text.charAt(i);
            if (c < 64) {
                buffer.append(TEXT[c]);
            } else {
                buffer.append(c);
            }
        }
        return buffer.toString();
    }

    // ---------------------------------------------------------------- enocde text

    public static String block(Object object) {
        if (object == null) {
            return EMPTY_STRING;
        }
        return block(object.toString());
    }

    /**
     * Encodes text into HTML-safe block preserving paragraphes. Besides the {@link #text(String) default
     * special characters} the following are replaced, too:
     * <ul>
     * <li>\n with <br></li>
     * <li>\r with <br></li>
     * </ul>
     * <p/>
     * <p/>
     * Method accepts any of CR, LF, or CR+LF as a line terminator.
     */
    public static String block(String text) {
        int len;
        if ((text == null) || ((len = text.length()) == 0)) {
            return EMPTY_STRING;
        }
        StringBuilder buffer = new StringBuilder(len + (len >> 2));
        char c, prev = 0;
        for (int i = 0; i < len; i++, prev = c) {
            c = text.charAt(i);
            if ((c == '\n') && (prev == '\r')) {
                continue; // previously '\r' (CR) was encoded, so skip '\n' (LF)
            }
            if (c < 64) {
                buffer.append(BLOCK[c]);
            } else {
                buffer.append(c);
            }
        }
        return buffer.toString();
    }

    // ---------------------------------------------------------------- encode text strict

    public static String strict(Object object) {
        if (object == null) {
            return EMPTY_STRING;
        }
        return strict(object.toString());
    }

    /**
     * Encodes text int HTML-safe block and preserves format using smart spaces.
     * Additionaly to {@link #block(String)}, the following characters are replaced:
     * <p/>
     * <ul>
     * <li>\n with <br></li>
     * <li>\r with <br></li>
     * </ul>
     * <p/>
     * This method preserves the format as much as possible, using the combination of
     * not-breakable and common spaces.
     */
    public static String strict(String text) {
        int len;
        if ((text == null) || ((len = text.length()) == 0)) {
            return EMPTY_STRING;
        }
        StringBuilder buffer = new StringBuilder(len + (len >> 2));
        char c, prev = 0;
        boolean prevSpace = false;
        for (int i = 0; i < len; i++, prev = c) {
            c = text.charAt(i);

            if (c == ' ') {
                if (prev != ' ') {
                    prevSpace = false;
                }
                if (prevSpace == false) {
                    buffer.append(' ');
                } else {
                    buffer.append(" ");
                }
                prevSpace = !prevSpace;
                continue;
            }
            if ((c == '\n') && (prev == '\r')) {
                continue; // previously '\r' (CR) was encoded, so skip '\n' (LF)
            }
            if (c < 64) {
                buffer.append(BLOCK[c]);
            } else {
                buffer.append(c);
            }
        }
        return buffer.toString();
    }
}
Enjoy while it lasts!