Monday 21 November 2011

Certificate based authentication on Linux

You can create your own certificate and use it in order to authenticate on Linux machines while connecting through SSH. But be aware that SSHD (SSH daemon is the one that accepts SSH connections, and can be perceived as SSH server) is very sensitive to file permissions. If you set wrong permissions on .ssh or .ssh/authorized_keys, your certificate won't work. Here are the correct file permissions (considering that you're in home folder of remote machine target user):

$ chmod 0700 .ssh
$ chmod 0600 .ssh/authorized_keys

Wednesday 16 November 2011

Retrieving large resultsets from postgres using psql

Consider that you have a large table in database and you want to export it to a CSV file. What's your first idea? I started using PgAdmin - wrote a simple SELECT and ran "Execute to file". But it gave me out "out of memory for query result". It turns out that PgAdmin tries to load the resultset into memory (client side), but fails since memory limitations. Then I tried psql with same results. Then I found really good variable named FETCH_COUNT in http://www.postgresql.org/docs/9.1/static/app-psql.html and here's how I used it:

statistics=#\a
statistics=#\f ,
statistics=#\set FETCH_COUNT 500000
statistics=#select * from MY_HUGE_TABLE
statistics-#\g /home/arvids/statistics.csv
  1. Disable aligned mode, because we don't want extra whitespaces in CSV file
  2. Then set field separator to comma (it's a CSV file after all)
  3. Then set fetch count to 500000 so that client (my computer) could load the partial resultset into memory
  4. Now execute query
  5. ... and route it to a file

Friday 7 October 2011

Upgrading postgres from 8.4 to 9.1 and LOB permission problems

Today upgraded to new Ubuntu 11.10 and a new version of postgresql came with it. There was some minor problems regarding database migration from 8.4 to 9.1.

Migrating isn't so easy as I would like it to be, but it's doable:

  • You'll need running postgresql 8.4 (old version)
  • From command line, run "pg_dumpall -h localhost -p 5432 -U postgres -W > pg.backup"
  • Now you can uninstall the old version and configure the new version (9.1) to use 5432 port by editing /etc/postgresql/9.1/main/postgresql.conf
  • Now run "sudo -u postgres psql -f pg.backup postgres" (this will import the backup file)

This is the point where everything seemded fine, but it wasnt, because one select showed me error: "ERROR: permission denied for large object xxxxx". In order to solve this problem, this forum helped me where I found this little helpful script that changes the owner of large objects:

do $$
declare r record;
begin
for r in select loid from pg_catalog.pg_largeobject loop
execute 'ALTER LARGE OBJECT ' || r.loid || ' OWNER TO owners_username';
end loop;
end$$;

Connect to each database as 'postgres' and execute this query. It will change all permissions of large objects in that database by setting an owner to the specified value. Btw, list all large objects you can with using these console commands:

sudo -u postgres psql postgres
postgres=# \connect portal4
You are now connected to database "portal4" as user "postgres".
portal4=# \lo_list

In order to connect to postgres using pgadmin, I had to change the postgresql password (see http://doginpool.blogspot.com/2009/06/adding-local-postgres-database-to.html)

Friday 23 September 2011

Reading XML files in java

Example description: I have to create a custom group store so I could add them to the groups retrieved from LDAP. My choice is storing them in XML files. So I thought someone might find my experience interesting.

Prerequisites: Eclipse for java development with m2e plugin.

Create XSD file

Why? This will make creating XML files very easy and will ensure that those files will be readable as expected. I won't go into details, since it's not so hard to create even very complex schema definition files. But here's the result (file /src/main/resources/lv/rtu/itd/vivs/user-groups-1.0.xsd):

<schema elementformdefault="qualified" targetnamespace="http://vivs.rtu.lv/schemas/user-groups" xmlns:tns="http://vivs.rtu.lv/schemas/user-groups" xmlns="http://www.w3.org/2001/XMLSchema">

  <element name="user-groups" type="tns:user-groups-type">

  <complextype name="user-groups-type">
    <sequence>
      <element maxoccurs="unbounded" name="user-groups" type="tns:user-group" />
    </sequence>
  </complextype>

  <complextype name="user-group">
    <sequence>
      <element name="uid" type="string" />
      <element maxoccurs="unbounded" name="group" type="string" />
    </sequence>
  </complextype>

</element>

Configure class generation

Create file src/main/binding/bindings.xjb (this will instruct the generator how to create new classes):

<bindings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xmlns:xs="http://www.w3.org/2001/XMLSchema" 
  xmlns="http://java.sun.com/xml/ns/jaxb"
  xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc" 
  xsi:schemaLocation="http://java.sun.com/xml/ns/jaxb http://java.sun.com/xml/ns/jaxb/bindingschema_2_0.xsd"
  version="2.1">

  <bindings>
    <globalBindings>
      <serializable uid="1" />
      <xjc:simple />
    </globalBindings>
  </bindings>

  <bindings schemaLocation="../resources/lv/rtu/itd/vivs/user-groups-1.0.xsd">
    <schemaBindings>
      <package name="lv.rtu.itd.vivs.xml.user_groups" />
    </schemaBindings>
  </bindings>

</bindings>

Configure pom.xml and add something like this in order to tell Maven that it must now generate classes using jaxb2:

  <build>
    <plugins>
      <plugin>
        <groupId>org.jvnet.jaxb2.maven2</groupId>
        <artifactId>maven-jaxb2-plugin</artifactId>
        <version>0.7.4</version>
        <executions>
          <execution>
            <id>generate-xjc-sources</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <bindingDirectory>src/main/binding</bindingDirectory>
          <schemaDirectory>src/main/resources</schemaDirectory>
          <schemaIncludes>
            <include>lv/rtu/itd/vivs/*.xsd</include>
          </schemaIncludes>
          <extension>true</extension>
          <args>
            <arg>-no-header</arg>
            <arg>-readOnly</arg>
            <arg>-mark-generated</arg>
          </args>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>com.sun.xml.bind</groupId>
            <artifactId>jaxb-xjc</artifactId>
            <version>2.2.1</version>
          </dependency>
        </dependencies>
      </plugin>
    </plugins>
  </build>

Now, probably, you must click on project and choose Maven > Update Project Configuration

Java code

Create interface for class:

package lv.rtu.itd.vivs.service;

import java.util.Set;

public interface GroupService {
    Set getGroups(String uid);
}

Create class that will unmarshal (read) the XML document:

package lv.rtu.itd.vivs.service.impl;

import java.io.IOException;
import java.io.InputStream;
import java.util.HashSet;
import java.util.Set;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.io.ClassPathResource;

import lv.rtu.itd.vivs.service.GroupService;
import lv.rtu.itd.vivs.xml.user_groups.UserGroup;
import lv.rtu.itd.vivs.xml.user_groups.UserGroups;

public class XmlGroupService implements GroupService {
    
    private Logger logger = LoggerFactory.getLogger(getClass());
    
    private UserGroups groups;
    
    public void setFilename(String filename) {
        JAXBContext ctx;
        try {
            ctx = JAXBContext.newInstance("lv.rtu.itd.vivs.xml.user_groups");
            Unmarshaller unmarshaller = ctx.createUnmarshaller();
            InputStream is = new ClassPathResource(filename).getInputStream();
            this.groups = (UserGroups) unmarshaller.unmarshal(is);
        } catch (JAXBException e) {
            logger.warn(e.getMessage(), e);
        } catch (IOException e) {
            logger.warn(e.getMessage(), e);
        }
    }
    
    @Override
    public Set getGroups(String uid) {
        for (UserGroup group : groups.getUserGroups()) {
            if (group.getUid().equals(uid)) {
                return new HashSet(group.getGroups());
            }
        }
        return new HashSet();
    }
}

Create test class (under src/test/java):

package lv.rtu.itd.vivs.service;

import java.util.HashSet;
import java.util.Set;

import lv.rtu.itd.vivs.service.impl.XmlGroupService;

import org.junit.Assert;
import org.junit.Test;

public class XmlGroupsTester {
    
    @Test()
    public void testGetGroups() {
        XmlGroupService service = new XmlGroupService();
        service.setFilename("user-groups.xml");
        
        Set expected = new HashSet();
        Assert.assertEquals(expected, service.getGroups("not-existing-user"));
        
        expected.add("my/demo/group");
        expected.add("my/demo/group2");
        expected.add("my/demo/group3");
        Assert.assertEquals(expected, service.getGroups("user.demo"));
    }
}

Now create XML file (src/test/resources/user-groups.xml:

<?xml version="1.0" encoding="UTF-8"?>
<user-groups xmlns="http://vivs.rtu.lv/schemas/user-groups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://vivs.rtu.lv/schemas/user-groups http\://vivs.rtu.lv/schemas/user-groups">
  <user-groups>
    <uid>user.demo</uid>
    <group>my/demo/group</group>
    <group>my/demo/group2</group>
    <group>my/demo/group3</group>
  </user-groups>
</user-groups>

Now you can run tests and check the results.

Thursday 1 September 2011

Eclipse indigo minimal install

Since i've found out about installation of customized (minimal) eclipse, I haven't ever used full version of PHP, JEE or PDT packages. Here's a bit updated version:

In order to install eclipse that suits to your needs, you must:

  1. Go to Eclipse downloads page
  2. Select Other Downloads near Eclipse Classic
  3. I couldn't find a minimal installation for Eclipse 4.0 or 4.1, hence you must switch to 3.x downloads page
  4. Next select version of your choice
  5. Scroll down to "Platform Runtime Binary" and download the package depending on your OS and architecture
  6. Uzip and start Eclipse
  7. Select Help > Install New Software ...
  8. And install appropriate packages:
    1. For Java web development I use:
      • Eclipse Java EE Developer Tools
      • Eclipse Java Web Developer Tools
      • Eclipse Web Developer Tools
      • All JST Server Adapters (2 packages), JST Server UI, WST Server Adapters (this is the part where it's hard to guess where Tomcat server adapter is located, if you want to use Tomcat installation from Eclipse)
      • m2e - Maven Integration for Eclipse + integrations for WTP, SCM, etc.
      • Subclipse + Adapters
      • Spring IDE: Core, Web Flow, Autowire Extention, Security Extention
    2. For PHP web development I use:
      • PHP Development Tools (PDT) SDK Feature
      • Subclipse + Adapters
Note that both installations indirectly include support for Javascript, CSS, HTML and other file editing. And this is MY way of doing things - you're not restricted to these packages - feel free to add any additional plugins of your choice.

If you have any suggestions or questions, feel free to comment. Blogging and commenting is all about knowledge exchange (that is fantastic!).

Thursday 11 August 2011

Eclipse doesn't shut down the process after exit

Another issue that I solved today was regarding closing Eclipse. After migrating to 64bit Ubuntu, my Eclipse (newly installed) didn't want to close. After closing, eclipse continued to run in background. In fact when I noticed it, I had 3 or 4 running eclipse processes since plugin installation requires restarts and each restart added one process eating all swap space. There was no evidence in GUI of running processes, but process manager did show them running. Then i came across this bug in launcpad and it did solve the problem.

Solution: Disable assistive technologies: System > Preferences > Assistive Technology Preferences > "Enable assistive technologies"

Dual monitors in LXDE

Trying out LXDE as my desktop environment on ubuntu box (Gnome is getting easy to use, but too heavy). There were some problems with dual monitor support (couldn't find a config app), but this line solved everyting:

xrandr --output VGA-0 --auto --left-of DVI-0

Inspired by this blog post.

Friday 15 April 2011

I18n portlets

Supporting multilingual content is very important in my organization. Unfortunately not everything is so easy as it might seem. Today I solved a little problem which was bugging probationer who is under my guidance. As it turned out, there were two major problems with this tutorial:

  1. He was using wrong fmt.tld file. The best thing is to define taglib using this line (notice the difference from line in tutorial):
    <%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %>
  2. Ssecond pitfall was that uPortal implementation which we use, didn't set locale automagically. If your portal implementation isn't setting this attribute (with key 'javax.servlet.jsp.jstl.fmt.locale'), then use these lines (the first one is needed in order to define renderRequest value):
    <portlet:defineObjects />
    <fmt:setLocale value="${renderRequest.locale}"/>

Ok, now putting all together if you need to translate content in JSP:

  1. Check that your portlet supports locales of your choice
    <portlet>
    ...
    <supported-locale>en</supported-locale>
    <supported-locale>lv</supported-locale>
    ...
    </portlet>
  2. Create files named messages.properties and messages_lv.properties in package com.example with key-value pairs of your choice. For example:
    messages.properties:
    welcome=Welcome to my page!

    messages_lv.properties:
    welcome=Esiet sveicināti manā mājas lapā!
  3. Add standard library to classpath of your web application (i.e. in WEB-INF/lib) and define use of formatting taglib in JSP page, specify locale, and translate:
    <%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %>
    <%@ taglib prefix="portlet" uri="http://java.sun.com/portlet" %>
    ...
    <portlet:defineObjects />
    <fmt:setLocale value="${renderRequest.locale}"/>
    <fmt:setBundle basename="com.example.messages"/>
    <h1><fmt:message key="welcome"/></h1>

Hope this helps

Thursday 7 April 2011

Trusting self signed certificate in java

Simple version of how to trust self-signed certificates.
$ wget https://raw.githubusercontent.com/escline/InstallCert/master/InstallCert.java
$ javac InstallCert.java 
$ java InstallCert host:port
$ cp jssecacerts /usr/java/jdk1.6.0_21/jre/lib/security/

restart JVM and you're done. Of course, you can always choose longer path and follow JSSE documentation

Thursday 24 March 2011

Trusting self-signed certificates

Yesterday I worked on a task that involved changing user password in Active Directory. Changing users pasword in AD requires secure connection (636 port) and self signed certificate was used (it was a test machine after all). I knew that Java won't accept that certificate, but was surprised that even Luma didn't accept the certificate, hence started looking over google to find a solution. Surprisingly nothing came out. There were tons of materials describing how to create a self signed certificate, but nothing how to accept one. After a couple of hours I remembered that one internal project documentation described the thing i needed. Steps:


  1. openssl s_client -connect ad.example.com:636
  2. Copy the certificate (everything between lines 'begin certificate' and 'end certificate' including both lines)in file (e.g. ad-example-com.cert)
  3. keytool -import -alias ad-example-com -file ad-example-com.cert -keystore ad-example-com.keystore
  4. Add certificate and keystore file to Luma (in server configuration view) and now you should be able connect to AD without problems...

In order to connect to AD using Java and accept SSL connection, use see this blog entry (by Andreas Sterbenz). InstallCert java file attached to his blog post is very easy to use in order to create jssecacerts file which can be copied to $JAVA_HOME/jre/lib/security in order to accept the self signed certificate.

Monday 14 March 2011

Hibernate and native SQL fetching

Today it took me about 3 hours to find a problem in my code that was caused by HHH-2831. Unfortunately Hibernate documentation is vague regarding 'return-join' elements. By reading comments of bug, i realized that i'm not the only one facing the same problem and in all cases it was intuition of developers which proved to be wrong.

I'll explain briefly the essence of problem. Consider two simple hibernate class mappings and named native SQL query:

  <class name="ObjectA" table="OBJECT_A">
<id column="ID" name="id">
<generator class="native" />
</id>
<property name="title" column="TITLE" type="string" length="255" not-null="true" />
<list name="children" cascade="all-delete-orphan">
<key column="OBJECT_A_ID" not-null="true" />
<list-index column="ORDER_INDEX" base="0" />
<one-to-many class="ObjectB" />
</list>
</class>
<class name="ObjectB" table="OBJECT_B">
<id column="ID" name="id">
<generator class="native" />
</id>
<property name="title" column="TITLE" type="string" length="255" not-null="true" /> </class>
<sql-query name="getAll" read-only="true">
<return alias="a" class="com.example.ObjectA" />
<return-join alias="b" property="a.children" />
select {a.*}, {b.*}
from OBJECT_A a
inner join OBJECT_B b on b.OBJECT_A_ID = b.ID
</sql-query>

Now... what do you expect of when executing something like:

List result = session.getNamedQuery("getAll").list()

I was expecting it to be a list of objects of class ObjectA without any duplicates. Unfortunately this is not the case (like it would've been when using HQL). In fact it returns a list of Object[2] elements and with as many duplicates of ObjectA as it has children. The second object in array is ObjectB (child) object. Hence you must filter the list for duplicate elements:

        List finalResult = new ArrayList();
for (Object[] objs : result) {
ObjectA obj = (ObjectA) objs[0];
if (!finalResult.contains(obj)) {
finalResult.add(obj);
}
}

I wonder what will happen in case of another level... They cannot be filtered if those collections are not inverse, because it would automatically update the list of children (remove them).

Thursday 10 February 2011

UTF-8 encoding with nusoap

Again problems with UTF-8 encoding. This time it's related to nusoap. I hate when developers assume that everyone uses english by default. Everyone should use UTF-8 by default. But now to the point...

When submitting a request to nusoap-based web service (nusoap_server), it assumes that characters ar encoded using ISO-8859-1. In fact very first search made by Google with keywords "nusoap utf-8" showed me the right path, but since this is my notebook, i'm writing down to remember: When creating web service, you must tell nusoap to not decode utf8 since encoding is utf8 already (see code snippet below).


$server->configureWSDL('myWebServiceMethod', $namespace);
$server->wsdl->schemaTargetNamespace = $namespace;
$server->decode_utf8 = false;