Friday, July 22, 2011

GRASS: Day 5

Catching up on yesterday's work - work meeting in the evening two nights in a row does challenge my blog writing skills... -

I am finally seeing some good progress in my understanding of the raster function by starting to find useful tricks... but I also came to the conclusion that I might need to be more selective in my reading because trying to understand every single page might bring me to a similar status and appearance to Mathusalem.

Again I witness today that the Python GUI (still the best thing for me to use as I couldn't get a proper display with the command line) has some command that it doesn't take however there seems to be usually alternative.

A great example of this is with the querying function for raster layers.
Mitasova and Neteler recommend the following command which enables a interactive querying methods with controls... which fails in the python GUI
d.what.rast
The manual option (again manual in my world means to option that you have to click on in the GUI) has been highlighted in circle in the screenshot below ( I had to draw a equivalent to the pointer)...

 but is it also possible to still do it through the command line with the following code (with North Carolina dataset, as usual) with the coordinates of the point.
r.what -f landuse96_28m east_north=638650.220610
(the -f flag includes the category's name)

now some other informative query codes
#numeric terrain profile
r.profile
#univariate statistic can use flag -e to show median and 90th percentile
r.univar
#statistic report, possible to use unit to ask for extra info eg. cell, hectares, percent
r.report landuse96_28m unit=c,h,p

Solution to Day 4's Problem 1

now I hope you are all well seated because... I have found the solution to yesterday problem 1, ouf, how exciting... (sorry with perspective, thinking how chuffed I was yesterday to figure that one, well, it is fairly worrying...)

So as a reminder, hell were breaking loose when  was typing this line of code...
d.rast -o elevation val=100-160
but I could do the overlay manually and in the menu they were even saying how overlay was the flag o...
well due to a mixture of desperation, genius or insanity, since I was facing a similar code problem... (sorry didn't take note of the code itself, but as you will soon see it is irrelevant)
I tried this instead.
d.rast elevation val=100-160 -o
It worked. In vernacular, I moved -o to the end, since I noticed that some code had the flag at the end and when I was doing so the GUI was giving me the o option... (which is why I like this python GUI, we are maintaining a lovely relationship now)

The quest of the missing code

A little problem I encountered today though was a missing code... Not only it didn't work in the GUI, it even wasn't in the list... and when I looked into the GRASS manual, well it was none existant... So if anyone can explain why they make quite a fuss about it in the book (maybe it only works and even show up to a chosen few) please let me know... hehe.
r.digit
(v.digit exists, but not r.digit)

But let's leave these myterious matters behind, to expose the first concrete useful thing for a archaeologist I have finally achieved on GIS...
Can you guess? Yes... That's it.
I have generated a random sampling map... WONDERFUL. (now that is exciting and even useful)
r.random.cells
Here is the masterpiece... Isn't it absolutely gorgeous? And I have even modified the Metadata about it...
Un vrai chef-d'oeuvre...

Thursday, July 21, 2011

GRASS: Day 4

Quick summary of yesterday, similar to day 3, lots of reading, less codes and a lot of headaches...

I've completed Neteler and Mitasova ch.4 about uploading files. I am hoping at this early stage to stick to more common files types e.g. GeoTIFF and Esri Shapefile, but I tried to get a least a grasp of what else could be achieved. A lot of the work seem to be done with external tools Gdal for raster layers and OGR for vector (which is visible in the command r.in.gdal). I've touched these tools before as they are also part of Quantum. They enable among other things to only upload a portion of a large raster to match the selected region etc.

I am now starting on Raster layers...
After a bit of playing around, I started to face difficulties - if you want to crash the software like me, follow the instructions below -
As I have mentionned on Day 3, I'm currently sticking to the python GUI as it is the only one which display information correctly and it has good support when writing code... But. the GUI does not respond to all the codes given in the book (which is based on using command line in a UNIX system).
For example Problem n.1, let's try this (with the North Carolina sample data).
g.region rast=elevation -p # that line is remembered from the top of my #head, it might be not perfect but it works for this example
d.rast elevation # this line is optional, the problem occurs with and #without
d.rast geology_30m
d.rast -o elevation val=100-160

now in theory this should overlay the elevation values between 100-160 on top of the geology layer... but it doesn't. Here is an quick image of what the code of the problem looks like.
and for some reason, it seems to jam after this...






















If I remove the overlay command (-o) it works, but I then need to do the overlay manually (by going in layer/properties in the Map layers window).

For some curious reason the same type of problem code come when I try:
d.rast help
which I would have thought is a pretty standard request.

I do not know where I have gone wrong yet, despite a lot of figgeting about. I will have to try again later.

Another difficulty (Problem n.2) came when I was trying to apply a legend, north arrow etc. It seems that the GUI does not accept to use command console to add text to a map for example.

Still I have succeeding in having fun with a few simple raster commands, for e.g.
r.info
r.colors

I have learned how to manipulate and modify the metadata information with
r.info
r.category
r.support (to do modifications)

and most importantly, I must admit I am absolutely amazed at the quality of the help files (in comparison of Quantum, at least for the code section) Here is a sample of how clear their help is (which for the poor Neophyte that I am is quite vital).
Isn't this absolutely wonderful?

Tuesday, July 19, 2011

GRASS: Day 3

Here I am back to it... it..., sounds slightly alien... an unknown organism, ready to give you access to its great powers, if you crack its logic and language.

Well with the conclusions and explorations of Day 2, I have decided to work with the python GUI, as it is the only one which display 2D maps (even if I still need a manual touch to make them look as they should). Also, now that I understand why some command like d.mon x0 are not working since they don't apply to the GUI, I can more appreciate the help that the command lind does offer e.g. any failure is detailed and solutions are often offered, I think I can also visualise the code that I do manually (by pressing icons rather than entering command line) - need to triple check that last aspect though, I know it applies when the code fails -.

Mainly what I did wasn't that interesting, I mostly read about the project data management (that Grass does by itself) work with coordinate systems, how to start new projects (not worth describing, the easiest way to learn is to use the GUI and have a look around), projection with the help of Gdal and OGR as well as uploading files.

but MAINLY, I try to solve a few issues I faced on Day 1 with uploading simple files.

Here are the details.
On Day 1, before playing much with the book (IMPORTANT FACT: I am currently using to help my learning of Grass NETELER, M. and H. MITASOVA Open Source GIS A Grass Approach 3rd Edition http://www.amazon.co.uk/Open-Source-GIS-GRASS-Approach/dp/1441942068/ref=sr_1_1?ie=UTF8&qid=1311109845&sr=8-1 ) I had try to upload some simple Ordinance Survey shapefiles and raster (GeoTiff to be exact) that I knew well. And I failed... miserably, which is why I simply started to work with the North Carolina sample data.

But today, with new determination, I tried again. The command line gave me the following mistake "Grass 'ascii' codec can't encode character u'\xe9' in position 37: ordinal not in range(128)" (note the position changing depending on the file). Obviously I couldn't make much sense of what that meant. After a browse online, I started to get it was a notion of character (a frequent problem with my french canadian keyboard) that wasn't accepted properly. I remembered about my mentor (yes because you see, GIS is a bit like Jedi, there are mentors -- I won't go up to say master-- and padawan) about the location of files and folders and how simple name without white spaces are best. So just for the fun of it, I moved the location of the files I was trying to open closer to C:/ and I cleaned the names... Well guess what? It worked.
KEY TIP: keep the files you want to use in a simple folder e.g. C:/mygis and keep the name without accents and spaces.

Now that this problem was solved, were the files working and displaying the lovely Cumbria? No, of course not. It couldn't be that simple.
I started to hit another problem which had plagued me in Qgis, and left me on the verge to give up: projection unknown and no match to the projection of the project/location.
This time the help in the command line offered me a solution (which I later found also mentionned in the book): override the projection check with -o. The best thing to do (even if you tried to upload the file manually) is to go fetch in the command history the piece of code, add -o after the first portion of the code r.in.gdal (for raster files) and the projection of the location/project will be applied to the layer.
Now why does that happen? Following Neteler and Mitasova (the book people) it is a common problem with data in TIFF/TFW format: the tfw file contains the coordinate but no projection information.
So armed with that knowledge, I was able to also apply to override to shapefiles and finally managed to get my Ordinance Survey data uploaded (I tested the process with some data given by my mentor as well).
Ordinance survey data from Cumbria, square NY and A roads

So mainly I read - not worth trying all the codes for subjects mentionned until needed - and solved an annoying problem. It was maybe a small step (and I am not even sure if I could apply anything similar in Qgis) but at least it is progress.

And as Joseph Joubert said "The aim of an argument or discussion should not be victory, but progress."

Wednesday, July 13, 2011

GRASS: Day 2

Where is Day 1 gone? Well I did say I am a rubbish blogger, moreover I had no journal yesterday...

Some basics, but then the boring bits starts.
more information on Grass can be found here... http://grass.osgeo.org/ It is less user friendly than Quantum, but it is a lot more powerful and certainly more stable, as it has been around for many years (if I read well it originates from military uses, how strange is it that a lot of the world greatest inventions where born out of war, what image does it give of human kind?)
I downloaded it from OSGeo4W, from which you can get many open source software in one installer, I've been told that is better to do it that way, as it gives also a command line post to start all these programs (and maybe they can interact better with each other, who knows...). Bref, (yes another french word worth knowing, it sounds better than "anyway" and is smaller than "to summarise") here is a link. http://trac.osgeo.org/osgeo4w/

Now to avoid anyone as silly as me to figure out how to get grass to work from the command line interface, here you go, rince your eye (and look in Usage): http://trac.osgeo.org/osgeo4w/wiki/pkg-grass

Boring bits starting... now.

After a lots of mixed results and fumbling yesterday here is my conclusions for day 2.

I understand fairly well how the Grass command system work now after a couple of days of playing with it,  I am very impress that there is help and index for all the commands (something that I did not find for Quantum and the python language and made my life close to a living hell)
here is the code:
g.manual -i (then you go in Display commands manual, then at the bottom of the page in full index)

Here are the main pieces pieces of code I've played with (to use with the North Carolina Sample data)
#list maps
g.list rast
g.list vect

#normal 2d display
g.region rast=elevation -p
d.erase #does not work in the GUI, I wonder if it mainly apply for UNIX
d.rast elevation
d.his h=elevation i=elevation_shade
d.vect streams col=blue
d.vect roadsmajor
d.vect overpasses icon=extra/bridge size=15 fcol=blue

#and the most impressive results with the 3D
nviz elevation vect=streams,roadsmajor point=overpasses

#and that is it, it sounds small but I had quite a headache to just get all of that to work the way I wanted

I have played a bit with the command line from the python interface of Grass, it is good as it gives you choices when you are typing, however, if I manage to display a map, the layers options that I had decided, e.g color blue, etc, do not show up until I manually go check the options in their propriety (right click or double click) in the display of map layers.

When I finally got how to use the OSGeo 4W command line interface (see above and yes I do feel relatively stupid) I tried to display map, but it comes as a png of a single layer at the time on my desktop, tried to open a monitor with
d.mon x0
but that is just for Unix or Mac, otherwise they recommend to download a X-server (not quite sure which one to get) or to use another GUI (I really hate this name, it sounds slimy) with:
gis.m
But the blooming GUI does seem to want to display my map...

Still all is not lost, I had a bit of a play around with the nviz module for 3D (for now it is more 2.5D but nevermind) and at least I seem to be able to get results there...

Here is a Screenshot.
nviz displaying North Carolina sample data

Confessions of a failed blogger

Yes... I admit it...

I am rubbish, utterly rubbish at keeping a blog.

A very good thing that I am not trying to become a writer...

My poor computer is filled with a few nearly finished pieces about GIS, which seem doomed to a life in limbo without ever reaching the CLOUD... they were obstructing too much my mind and distracting me from the chaotic path to the GIS Open source enlightenment... & let me say firm and loudly how chaotic this path is... in April and May I got battered and bruised, between unfinished help index, and limited tutorials, I thought many times to die and give up... yet a few months later, I am still here...

And what I need... Is a journal!!?
A journal so my kind-hearted mentor can follow what I'm trying to do - or failing often -. A journal so I, and you, if anyone tries to read it, can keep track of the happy days when it works, and the less happy and more common ones, when all fails... But beware, it is another attempt at writing, no promises that I will be better this time at completing it... and moreover, it will be dry, dry and boring. (although to warn you it will always start by the title of the project I am working on, then the day)  What will it be about? At the moment two distinct efforts... The first is to learn how to use GRASS, one of the most powerful open-source GIS software out there and especially its 3D functions http://grass.osgeo.org/. The second, the completion of the FG project (pronouce 'figi'), an interactive map of the Forest Gardens in Cumbria. I warned you but let me say it again, this is more to keep track than to attract litterate discussion, I'll try to add links and maybe if I'm inspired a couple of line of descriptions, but don't get your hopes up, don't hope for much.

Still amateurs of my witty interpretations of the keys to the GIS world, fear not... I do intend to take again my cybernatic pen and finish and write more of these litterary pieces more worthy of your attention than my journal ramblings, I still want to try to support other pure and innocent souls which embarks on this dangerous, this erratic and barmy idea - but Oh so worth doing - of learning GIS on your own...

Tuesday, April 5, 2011

Map Quest, Laser Quest... no GIS Quest!!! In search of Open Source Software that fits the bill.


(Homemade definition) Open Source software, i.e. software created by a community of users and developers worldwide, which are free, with the details of the program available so keen programmers can even modify, customize and improve the software for their own purpose and the community.

It was clear from the start of this journey, that I would need free Open Source Software... Let’s be honest, I am a recently reformed student (synonym of being skint), a free and legal software would fit me like a glove (yes I am experimenting with English idioms here). Another reason is that, despite that a off-the-shelf GIS software (polished, user-friendly, mostly without bugs, but well, expensive) would be fit for a king, choosing OpenSource tools, means encouraging a worldwide community of shared knowledge, an central idea to the enlightenment movement in XVIII c. with enterprise like L’Encyclopédie de Diderot et d’Alembert. And since we are following the GIS path of Reason, well we better follow it to the end...

OpenSource software, yes... But now what?

If you are light me interested to the shine of GIS technology, there are quite a few good sites which gives you listings of software both desktop and web-based, to help you to make a more appropriate choice for what you want to achieve with GIS. Here are a few.

http://www.osgeo.org/ one of the main organisations for the promotion of GIS, on the right-hand side there is a menu “OSGeo projects” which are the list of software they are involved in creating... It is a good way to start as they indicate a quick summary of what each software do and its main features.

http://freegis.org/database/?cat=0&_ZopeId=16454070A41D88t652Y interesting list of softwares and dataset, you can use the top criteria for a more accurate research, note. Some of the links are dated.

http://www.maptools.org/ good and clear website with application divided in sections for easier research.

http://gislounge.com/free-gis-applications/ other interesting links with descriptions.

Unfortunately, as I started exploring this labyrinth of numerous options, I had a fit of pique... because I do not understand yet half of what they are on about!! Still gobbledy googly gook (I intend to use this expression in every single post I write) to me...

My final choice, approved by my employer at Spatial Consultants, was for now Quantum GIS (also known as QGIS), an OSGeo tool, chosen for two words: USER-FRIENDLY, which the poor neophyte that I am, is desperately in need of. Other important favourable points are that it links with the powerful GIS software GRASS, has a series of plugins, and some useful written user manuals and tutorials (for useful list of some manuals, consult http://www.qgis.org/en/documentation/manuals.html and http://www.qgis.org/en/documentation/workshops.html and http://www.qgis.org/wiki/Users_Corner

Now armed with these hefty manuals and tutorials, as I am fit as a fiddle, I embark on the arduous path of taming QuantumGIS.

Wish me luck... Et swing la bacaisse dans l’fond d’la boîte à bois...

GI what? Answers for troubled minds

Hope you had a peaceful week-end, mine was troubled by this simple question: What is GIS (Geographic Information System)? How can it be explained to you, me and everyone without womble vocabulary, painful, abstract descriptions, and leave us all with this starry gaze offered by a great feeling of enlightenment? 

Here is a some extracts of a piece I composed for this purpose on Spatial Consultants site. (Do not dread it still contains deathly epidemies,  attractive protagonists, corruption, accidents, money, everything your heart desires). 

 Mind you, I still fear there is a bit a Goobledy Googly Gook (I love this word), in the "What is GIS" section... hopefully as go deeper in the meanders of GIS, simpler words will come...

What does GIS means to me (and you)?

An introduction to GIS by Delphine Jasmin-Bélisle, young archaeologist currently intern at Spatial Consultants to learn the usefulness of GIS for Archaeological, Heritage Management and Community-based projects.

If you have reached this page during your exploration of Spatial Consultants’ services, this question is probably on your mind, and you are most certainly seeking answers in simple, clear words to give you a better grasp of GIS. As a newcomer to GIS technology, without any previous formal background in computer technologies, I have asked myself the same question and, probably in the same way as you, I am keen to discover how and if, GIS can be use in tangible terms with time, money and expertise restrictions. I am also keen to explore the pitfalls of GIS technology, as no new science comes without its problems.  It is keeping these questions in minds that I have started my research and readings, and in the following sections will try to answer them, in the most concrete and practical way possible.

The power of maps: an historical introduction to GIS use
In 1854, cholera, a lethal infection, was striking Britain with not a soul knowing where the disease came from (at the time, it was believed that similarly to the Plague, cholera was due to polluted “bad” air). A London physician, Dr. John Snow, wary of these absurd theories, decided to place on a map of the city the various outbreaks of the sickness. Quickly he was able to trace correlations between the outbursts of the disease and infected water pump, particularly the Broad Street pump in Soho. Using statistics drawn from the map of outbreaks to support his theory, he retraced the source up to a waterworks company who was acquiring its water from sewage-polluted section of the Thames. 

What is GIS technology?
As this short story demonstrates, maps utility goes far beyond the simple display of information. But what if you could combine the visual efficiency of a map to the immense powers of digital analysis and databases... Welcome to the World of GIS. 

In our lives, most problems/projects, have a geographic component, (in fact it is even a difficult exercise to find something that cannot be mapped), GIS (Geographic Information System) technology combines geographical data (e.g location), with other attributes and values (ethnic groups present, type of trees, number of occupants, type of electricity, anything really!). It can store this information, analyse it and present it in different formats: maps, tables, etc. GIS does not actually stand for a single tool but rather a wide range of systems capable of a wide range of tasks depending on the problem, whether it is the distribution of a selected phenomenon, the impact of a specific event, the relationship between two components (e.g. traffic and health problems) in a specific region, or many other types of problems/projects. The key that differentiates GIS from other analysis and map software is that it can process data from dissimilar sources: map, image, video, statistic, CAD, etc. 

GIS visual display function enables easier and faster problem-solving, as it simplifies abstract figures and features to convey effectively information. It also helps to discover complex relationships which would not have been visible through texts or tables and facilitate lateral or creative thinking. Its database capacities enable to reduce hand-based information, free staff to do more efficient tasks  and most importantly enable to process enormous amount of data, which would unfeasible manually. Finally its analysis capacities can offer experimentations impossible to do in the Real World, with time-space simulation, to see how components influence each other over time, or to get answers for complex questions involving a lot of data . In brief, GIS saves time, manpower and money through effective database inventory and facilitate decision-making, management of resources and presentation of information.

Examples of GIS in our daily lives
Let’s use three examples to show how GIS is integrated in our daily lives.

Mr. A is a truck driver. During his trips, he uses a SatNav or his computer to find the best way to reach his destination. As he is often in unknown locations, he also uses his computer or his I-phone to find the nearby Thai restaurants, as Mr. A is a fine diner.

Ms. B is what we could call a keen shopper, fan of vouchers and good offers; she has loyalty cards for all her favourite shops. However today she was out of luck, on her way to the best offers, her car broke down and she had to call for emergency services, they responded fast and rescue arrived within 15 minutes.

Mr. C is a census analyst. He uses to information of the 2011 UK Census, to find which area needs a better allocation of funds and facilities. But to take his mind of work, he is also a keen diver – Mr. C lives in Cornwall you see – and is involved in a project to map the coral formation around Penzance and evaluate how harmful could be the pollution from the local maritime industry.

In these three cases, GIS is at the heart of the actions taken. 

Mr. A is the user of very common GIS applications, which answer simple questions, like the quickest route between two points or map facilities and indicate a selection following simple criteria, e.g. Thai Restaurants. These tools, e.g. AA planner, Satnav, Google map, are used daily by a great proportion of the population, who often are not even aware it is GIS.

 In this example, Ms. B doesn’t use herself GIS tool s, but she unknowingly generate GIS data. Her loyalty cards generate “lifestyle data”, enabling the retailers to track her purchases and evaluate better the patterns and taste of shoppers in that area. This can help them to create a distribution network of vouchers, tailor the selection of products to follow the customer’s taste and even help predict the chance of success of a new shop in that area. Also when Ms. B car breaks down, the emergency services she calls use a GIS program to determine with mobile unit is the closest to her, with the right tools to offer support, to enable a quick response. 

Of our three examples, Mr. C is probably the only one who knows the word “GIS”. As a census analyst, Mr. C converts the information acquired into anonymous blocks regrouped by neighbourhood; he will then use GIS complex analysis tools to find how this information can be used to improve decision-making in both national and local government. At home, Mr. C uses free Open Source GIS software to map the corals formations he recorded using a GPS. He then uses a GIS simulation tool, to forecast how different types and amount of pollution could affect the corals in the future.

 These three examples are only very simple illustrations of some of the functions used by GIS, but they summarise some of the key aspects of GIS: 
  1. used in a variety of fields, from national government to local environmental project, passing by commercial retailers
  2. used to solve a wide range of problems, from simple mapping to complex analysis of geographic and none-geographic (e.g. lifestyle information) data 
  3.  used every day by the majority of the population, consciously or not. 
Price Tag
So far, everything that had been said may lead you to think GIS is a complex and expensive technology, unreachable for small-medium businesses with limited financial means. But – luckily – this is where you are wrong!!

Certainly, in the past few decades, GIS was less accessible as the costs of hardware (powerful computers), software and data, were high and the options limited. In 2005, the GIS software industry accounted for 1.8 Billion in annual sales (these numbers do not include the sales in the data industry) and these numbers have just been climbing ever since. This led to the creations of a series of tailored program for the particular tool you need from GIS technology, available off-the-shelf and without a need for a super performing machine. Therefore it is possible to select a specialised product specific to your needs for lesser cost.

But there are even better – and cheaper!! – solutions now available. Recent years have seen the development of Open source softwares, i.e. software created by a community of users and developers worldwide, which are free, with the details of the program available so keen programmers can even modify, customize and improve the software for their own purpose. Thousands of companies and community-based projects have now opted for this free solution which, with the correct guidance, gives them the liberty and facility to have a project/program tailored to their needs at minimum cost.

No matter whether you choose software off-the-shelf or Open Source software, the key is to shop around and mostly importantly be informed. Specialists like Spatial Consultants will help you to make the best choice, tailored to your needs and budget, in order to avoid time and money loss.

Et voilà! GIS sur un plateau d'argent juste pour vous...

-for the full article please visit http://www.spatialconsultants.com/gis-to-me