Titan学习笔记-初识

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/wulinshishen/article/details/52955865

Titan 是一个在服务器集群搭建的分布式的图形数据库,特别为存储和处理大规模图形而优化。集群很容易扩展以支持更大的数据集,Titan有一个很好的插件式性能,这个性能让它搭建在一些成熟的数据库技术上像 Apache CassandraApache HBase、 Oracle BerkeleyDB。插件式索引架构可以整合 ElasticSearch 和Lucene技术。内置实现 Blueprints  graph API,支持 TinkerPop所有的技术。


特性

1、支持不同的分布式存储层

    - Apache Cassandra (distributed)
    - Apache HBase (distributed)
    - Oracle BerkeleyDB (local)
    - Persistit (local)

2、可以更加数据集的大小和用户基数弹性扩展

3、分布式存储的复制,高容错性

4、支持很多字符集和热备份

5、支持 ACID 和 eventual consistency(最终一致性)

6、支持的索引

    - ElasticSearch
    - Apache Lucene

7、内置实现 TinkerPop graph API

    - Gremlin graph query language
    - Frames object-to-graph mapper
    - Rexster graph server
    -  Blueprints  standard graph API


安装部署

下载解压titan-1.0.0-hadoop2.zip,测试环境版本:hadoop-2.7.2 + hbase-1.2.3 + elasticsearch-1.5.2

删除titan-1.0.0-hadoop2/lib目录下的hadoop-core-1.2.1.jar,拷贝titan-hadoop-1.0.0.jar,titan-hadoop-core-1.0.0.jar到titan-1.0.0-hadoop2/lib目录

可以使用命令$bin/titan.sh start测试是否安装成功

编辑conf/titan-hbase-es.properties文件
storage.backend=hbase
storage.hostname=host-10,host-11,host-12
index.search.backend=elasticsearch
index.search.hostname=host-10
index.search.elasticsearch.interface=TRANSPORT_CLIENT 
index.search.elasticsearch.cluster-name=data
index.search.elasticsearch.client-only=true

Gremlin Shell测试,示例来自官方,在hbase,elasticsearch上都会生成一个titan库
$bin/gremlin.sh
gremlin>graph=TitanFactory.open('conf/titan-hbase-es.properties')
==>standardtitangraph[hbase:[host-115]]
gremlin>GraphOfTheGodsFactory.load(graph)
==>null
gremlin>g=graph.traversal()
graphtraversalsource[standardtitangraph[hbase:[host-115]], standard]
gremlin>saturn=g.V().has('name','saturn').next()
==>v[256]gremlin>g.V(saturn).valueMap()
==>[name:[saturn],age:[10000]]
gremlin>g.V(saturn).in('father').in('father').values('name')
==>hercules
gremlin>g.E().has('place',geoWithin(Geoshape.circle(37.97,23.72,50)))
==>e[a9x-co8-9hx-39s][16424-battled->4240]
==>e[9vp-co8-9hx-9ns][16424-battled->12520]
gremlin>g.E().has('place',geoWithin(Geoshape.circle(37.97,23.72,50))).as('source').inV().as('god2').select('source').outV().as('god1').select('god1','god2').by('name')
==>[god1:hercules,god2:hydra]
==>[god1:hercules,god2:nemean]
gremlin>hercules=g.V(saturn).repeat(__.in('father')).times(2).next()
==>v[1536]
gremlin>g.V(hercules).out('father','mother')
==>v[1024]
==>v[1792]
gremlin>g.V(hercules).out('father','mother').values('name')
==>jupiter
==>alcmene
gremlin>g.V(hercules).out('father','mother').label()
==>god
==>human
gremlin>hercules.label()
==>demigod
gremlin>g.V(hercules).out('father','mother')
==>v[1024]
==>v[1792]
gremlin>g.V(hercules).out('father','mother').values('name')
==>jupiter
==>alcmenegremlin>g.V(hercules).out('father','mother').label()
==>god
==>human
gremlin>hercules.label()
==>demigod
gremlin>g.V(hercules).outE('battled').has('time',gt(1)).inV().values('name').toString()==>[GraphStep([v[24744]],vertex),VertexStep(OUT,[battled],edge),HasStep([time.gt(1)]),EdgeVertexStep(IN),PropertiesStep
gremlin>pluto=g.V().has('name','pluto').next()
==>v[2048]
gremlin>g.V(pluto).out('lives').in('lives').values('name')
==>pluto
==>cerberus
gremlin>// pluto can't be his own cohabitant
gremlin>g.V(pluto).out('lives').in('lives').where(is(neq(pluto))).values('name')
==>cerberusgremlin>g.V(pluto).as('x').out('lives').in('lives').where(neq('x')).values('name')
==>cerberus
gremlin>// where do pluto's brothers live?
gremlin>g.V(pluto).out('brother').out('lives').values('name')
==>sky
==>sea
gremlin>// which brother lives in which place?
gremlin>g.V(pluto).out('brother').as('god').out('lives').as('place').select()
==>[god:v[1024],place:v[512]]
==>[god:v[1280],place:v[768]]
gremlin>// what is the name of the brother and the name of the place?
gremlin>g.V(pluto).out('brother').as('god').out('lives').as('place').select().by('name')
==>[god:jupiter,place:sky]
==>[god:neptune,place:sea]
gremlin>g.V(pluto).outE('lives').values('reason')
==>nofearofdeathgremlin>g.E().has('reason',textContains('loves'))
==>e[6xs-sg-m51-e8][1024-lives->512]
==>e[70g-zk-m51-lc][1280-lives->768]
gremlin>g.E().has('reason',textContains('loves')).as('source').values('reason').as('reason').select('source').outV().values('name').as('god').select('source').inV().values('name').as('thing').select('god','reason','thing')
==>[god:neptune,reason:loveswaves,thing:sea]
==>[god:jupiter,reason:lovesfreshbreezes,thing:sky]

Java API调用操作Titan
<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <hadoop.version>2.7.2</hadoop.version>
        <hbase.version>1.2.3</hbase.version>
        <titan.version>1.0.0</titan.version>
	</properties>
        <!-- Guava依赖包 -->
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>16.0.1</version>
        </dependency>
        
		<!-- Elastic Search依赖包 -->
		<dependency>
			<groupId>org.elasticsearch</groupId>
			<artifactId>elasticsearch</artifactId>
			<version>1.5.2</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
		</dependency>
		
        <!-- Hadoop依赖报 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-annotations</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- HBase依赖报 --> 
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-protocol</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-annotations</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-hadoop2-compat</artifactId>
            <version>${hbase.version}</version>
        </dependency>

        <!-- Titan依赖包 -->
        <dependency>
		    <groupId>com.thinkaurelius.titan</groupId>
		    <artifactId>titan-core</artifactId>
		    <version>${titan.version}</version>
		</dependency>
		
		<dependency>
		    <groupId>com.thinkaurelius.titan</groupId>
		    <artifactId>titan-cassandra</artifactId>
		    <version>${titan.version}</version>
		</dependency>
		
		<dependency>
		    <groupId>com.thinkaurelius.titan</groupId>
		    <artifactId>titan-hbase</artifactId>
		    <version>${titan.version}</version>
		</dependency>
		
		<dependency>
		    <groupId>com.thinkaurelius.titan</groupId>
		    <artifactId>titan-es</artifactId>
		    <version>${titan.version}</version>
		</dependency>
        
        <dependency>
            <groupId>com.thinkaurelius.titan</groupId>
            <artifactId>titan-hadoop</artifactId>
            <version>${titan.version}</version>
        </dependency>

        <dependency>
            <groupId>com.thinkaurelius.titan</groupId>
            <artifactId>titan-hadoop-core</artifactId>
            <version>${titan.version}</version>
        </dependency>
import java.io.File;
import java.util.Iterator;
import java.util.Map;

import org.apache.tinkerpop.gremlin.process.traversal.Order;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.structure.Direction;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;

import com.thinkaurelius.titan.core.EdgeLabel;
import com.thinkaurelius.titan.core.Multiplicity;
import com.thinkaurelius.titan.core.PropertyKey;
import com.thinkaurelius.titan.core.TitanFactory;
import com.thinkaurelius.titan.core.TitanGraph;
import com.thinkaurelius.titan.core.TitanTransaction;
import com.thinkaurelius.titan.core.attribute.Geoshape;
import com.thinkaurelius.titan.core.attribute.Text;
import com.thinkaurelius.titan.core.schema.ConsistencyModifier;
import com.thinkaurelius.titan.core.schema.TitanGraphIndex;
import com.thinkaurelius.titan.core.schema.TitanManagement;

public class GraphOfTheGodsFactory {

	public static final String INDEX_NAME = "search";

	public static TitanGraph create(final String directory) {
		TitanFactory.Builder config = TitanFactory.build();
		config.set("storage.backend", "berkeleyje");
		config.set("storage.directory", directory);
		config.set("index." + INDEX_NAME + ".backend", "elasticsearch");
		config.set("index." + INDEX_NAME + ".directory", directory + File.separator + "es");
		config.set("index." + INDEX_NAME + ".elasticsearch.local-mode", true);
		config.set("index." + INDEX_NAME + ".elasticsearch.client-only", false);
		TitanGraph graph = config.open();
		GraphOfTheGodsFactory.load(graph);
		return graph;
	}
	
	public static TitanGraph create() {
		TitanFactory.Builder builder = TitanFactory.build();
		builder.set("storage.backend", "hbase");
        builder.set("storage.hostname", "host-10");
        builder.set("storage.tablename", "titan");
        builder.set("index.search.backend", "elasticsearch");
        builder.set("index.search.hostname", "host-10");
        builder.set("index.search.elasticsearch.interface", "TRANSPORT_CLIENT");
        builder.set("index.search.elasticsearch.cluster-name", "data");
        builder.set("index.search.elasticsearch.client-only", "true");
		TitanGraph graph = builder.open();
		return graph;
	}

	public static void loadWithoutMixedIndex(final TitanGraph graph,
			boolean uniqueNameCompositeIndex) {
		load(graph, null, uniqueNameCompositeIndex);
	}

	public static void load(final TitanGraph graph) {
		load(graph, INDEX_NAME, true);
	}

	public static void load(final TitanGraph graph, String mixedIndexName,
			boolean uniqueNameCompositeIndex) {
		// Create Schema
		TitanManagement mgmt = graph.openManagement();
		final PropertyKey name = mgmt.makePropertyKey("name").dataType(String.class).make();
		TitanManagement.IndexBuilder nameIndexBuilder = mgmt.buildIndex("name", Vertex.class).addKey(name);
		if (uniqueNameCompositeIndex) nameIndexBuilder.unique();
		TitanGraphIndex namei = nameIndexBuilder.buildCompositeIndex();
		mgmt.setConsistency(namei, ConsistencyModifier.LOCK);
		final PropertyKey age = mgmt.makePropertyKey("age").dataType(Integer.class).make();
		if (null != mixedIndexName)
			mgmt.buildIndex("vertices", Vertex.class).addKey(age).buildMixedIndex(mixedIndexName);
		final PropertyKey time = mgmt.makePropertyKey("time").dataType(Integer.class).make();
		final PropertyKey reason = mgmt.makePropertyKey("reason").dataType(String.class).make();
		final PropertyKey place = mgmt.makePropertyKey("place").dataType(Geoshape.class).make();
		if (null != mixedIndexName)
			mgmt.buildIndex("edges", Edge.class).addKey(reason).addKey(place)
					.buildMixedIndex(mixedIndexName);
		mgmt.makeEdgeLabel("father").multiplicity(Multiplicity.MANY2ONE).make();
		mgmt.makeEdgeLabel("mother").multiplicity(Multiplicity.MANY2ONE).make();
		EdgeLabel battled = mgmt.makeEdgeLabel("battled").signature(time).make();
		mgmt.buildEdgeIndex(battled, "battlesByTime", Direction.BOTH,
				Order.decr, time);
		mgmt.makeEdgeLabel("lives").signature(reason).make();
		mgmt.makeEdgeLabel("pet").make();
		mgmt.makeEdgeLabel("brother").make();
		mgmt.makeVertexLabel("titan").make();
		mgmt.makeVertexLabel("location").make();
		mgmt.makeVertexLabel("god").make();
		mgmt.makeVertexLabel("demigod").make();
		mgmt.makeVertexLabel("human").make();
		mgmt.makeVertexLabel("monster").make();
		mgmt.commit();

		TitanTransaction tx = graph.newTransaction();
		// vertices
		Vertex saturn = tx.addVertex(T.label, "titan", "name", "saturn", "age", 10000);
		Vertex sky = tx.addVertex(T.label, "location", "name", "sky");
		Vertex sea = tx.addVertex(T.label, "location", "name", "sea");
		Vertex jupiter = tx.addVertex(T.label, "god", "name", "jupiter", "age", 5000);
		Vertex neptune = tx.addVertex(T.label, "god", "name", "neptune", "age", 4500);
		Vertex hercules = tx.addVertex(T.label, "demigod", "name", "hercules", "age", 30);
		Vertex alcmene = tx.addVertex(T.label, "human", "name", "alcmene", "age", 45);
		Vertex pluto = tx.addVertex(T.label, "god", "name", "pluto", "age", 4000);
		Vertex nemean = tx.addVertex(T.label, "monster", "name", "nemean");
		Vertex hydra = tx.addVertex(T.label, "monster", "name", "hydra");
		Vertex cerberus = tx.addVertex(T.label, "monster", "name", "cerberus");
		Vertex tartarus = tx.addVertex(T.label, "location", "name", "tartarus");

		// edges
		jupiter.addEdge("father", saturn);
		jupiter.addEdge("lives", sky, "reason", "loves fresh breezes");
		jupiter.addEdge("brother", neptune);
		jupiter.addEdge("brother", pluto);
		neptune.addEdge("lives", sea).property("reason", "loves waves");
		neptune.addEdge("brother", jupiter);
		neptune.addEdge("brother", pluto);
		hercules.addEdge("father", jupiter);
		hercules.addEdge("mother", alcmene);
		hercules.addEdge("battled", nemean, "time", 1, "place", Geoshape.point(38.1f, 23.7f));
		hercules.addEdge("battled", hydra, "time", 2, "place", Geoshape.point(37.7f, 23.9f));
		hercules.addEdge("battled", cerberus, "time", 12, "place", Geoshape.point(39f, 22f));
		pluto.addEdge("brother", jupiter);
		pluto.addEdge("brother", neptune);
		pluto.addEdge("lives", tartarus, "reason", "no fear of death");
		pluto.addEdge("pet", cerberus);
		cerberus.addEdge("lives", tartarus);
		// commit the transaction to disk

		tx.commit();
	}
	
	public static void query(TitanGraph graph) {
		GraphTraversalSource g = graph.traversal();  
		System.out.println(g.V().has("name", "hercules").next().value("name"));  
		System.out.println(g.V().has("name", "hercules").next().values("name", "age"));  
		Iterator<Object> iterator = g.V().has("name","hercules").next().values("name", "age");  
	    while(iterator.hasNext()){  
	    	Object object = iterator.next();  
	        System.out.println(object);  
	    }  
	    Vertex saturn = g.V().has("name","saturn").next();  
	    System.out.println(saturn);  
        //得到 saturn的孙子节点  
        System.out.println(g.V(saturn).in("father").in("father").next().value("age"));  
  
        GraphTraversal<Edge, Edge> a =  g.E().has("place", P.eq(Geoshape.point(38.1f, 23.7f)));  
        System.out.println(a);  
        while(a.hasNext()){  
            Edge e = a.next();  
            System.out.println(e.keys());  
            System.out.println(e.label());  
            System.out.println(e.outVertex().value("name"));  
            System.out.println(e.inVertex().value("name"));  
            System.out.println(e.value("time") + "  :  " + e.value("place"));  
        }  
        
        Vertex hercules = g.V().has("name","hercules").next();  
        System.out.println(g.V(hercules).out("mother","father").values("name"));  
        GraphTraversal<Vertex,Vertex> mF = g.V(hercules).out("mother", "father");  
        while(mF.hasNext()){  
            Vertex v = mF.next();  
            System.out.println(v.label() + "  :  " + v.value("name"));  
        }  
  
        System.out.println(g.V(saturn).repeat(__.in("father")).times(2).next().value("name"));  
  
        GraphTraversal<Vertex,Vertex> monsters = g.V(hercules).out("battled");  
        while(monsters.hasNext()){  
            Vertex monster = monsters.next();  
            System.out.println(monster.label() + "  :  " + monster.value("name"));  
        }  
  
        monsters = g.V(hercules).outE("battled").has("time",P.eq(1)).inV();  
        while(monsters.hasNext()){  
            Vertex monster = monsters.next();  
            System.out.println(monster.label() + "  :  " + monster.value("name"));  
        }  
        Vertex pluto = g.V().has("name","pluto").next();  
        
        GraphTraversal<Vertex,Vertex> liveInTartarusVertex = g.V(pluto).out("lives").in("lives");  
        while(liveInTartarusVertex.hasNext()){  
            Vertex vertex = liveInTartarusVertex.next();  
            System.out.println(vertex.value("name"));  
        }  
  
        GraphTraversal<Vertex,Vertex> liveInTartarusVertexNo = g.V(pluto).out("lives").in("lives").where(  
                __.is(P.neq(pluto)));  
        while(liveInTartarusVertexNo.hasNext()){  
            Vertex vertex = liveInTartarusVertexNo.next();  
            System.out.println(vertex.value("name"));  
        }  
  
  
        GraphTraversal<Vertex,Vertex> liveInTartarusVertexNot = g.V(pluto).as("x").out("lives").in("lives").where(P.neq("x"));  
        while(liveInTartarusVertexNot.hasNext()){  
            Vertex vertex = liveInTartarusVertexNot.next();  
            System.out.println("======"+vertex.value("name"));  
        }  
  
        GraphTraversal<Vertex, Map<String, Vertex>> brothers = 
        		g.V(pluto).out("brother").as("god").out("lives").as("place").select("god","place");  
        while(brothers.hasNext()){  
            Map<String,Vertex> map = brothers.next();  
            System.out.println(map);  
            for(Map.Entry<String,Vertex> entry:map.entrySet()){  
                System.out.println(entry.getKey()+" : "+entry.getValue().value("name"));  
            }  
        }  
  
        System.out.println(g.V(pluto).outE("lives").next().value("reason"));  
  
        GraphTraversal<Edge,Edge> reasons = g.E().has("reason").as("r").values("reason").is(Text.textContains("loves")).select("r");  
        System.out.println(reasons);  
        while(reasons.hasNext()){  
            Edge e = reasons.next();  
            System.out.println(e.keys());  
            System.out.println(e.label());  
            System.out.println(e.value("reason"));  
        }  
  
        GraphTraversal<Edge,Map<String,Object>> reasons2 = 
        		g.E().has("reason").as("source").values("reason").is(Text.textContains("loves")).as("reason").select("source")  
                	.outV().values("name").as("god").select("source").inV().values("name").as("thing").select("god","reason","thing");  
  
        while(reasons2.hasNext()){  
            Map<String,Object> map = reasons2.next();  
            System.out.println(map);  
            for(Map.Entry<String,Object> entry:map.entrySet()){  
                System.out.println(entry.getKey() + " :  " + entry.getValue());  
            }  
        }  
	}

	/**
	 * 
	 * Calls {@link TitanFactory#open(String)}, passing the Titan configuration
	 * file path
	 * which must be the sole element in the {@code args} array, then calls
	 * {@link #load(com.thinkaurelius.titan.core.TitanGraph)} on the opened
	 * graph,
	 * then calls {@link com.thinkaurelius.titan.core.TitanGraph#close()}
	 * and returns.
	 * <p/>
	 * This method may call {@link System#exit(int)} if it encounters an error,
	 * such as
	 * failure to parse its arguments. Only use this method when executing main
	 * from
	 * a command line. Use one of the other methods on this class (
	 * {@link #create(String)}
	 * or {@link #load(com.thinkaurelius.titan.core.TitanGraph)}) when calling
	 * from
	 * an enclosing application.
	 * @param args
	 *            a singleton array containing a path to a Titan config
	 *            properties file
	 */
	public static void main(String args[]) {
//		if (null == args || 1 != args.length) {
//			System.err.println("Usage: GraphOfTheGodsFactory <titan-config-file>");
//			System.exit(1);
//		}
//		TitanGraph g = TitanFactory.open(args[0]);
//		load(g);
//		g.close();
		
		TitanGraph graph = GraphOfTheGodsFactory.create();
		load(graph);
		query(graph);
		graph.close();
	}
	
}

注意:
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator  
HBase与Guava版本不匹配,这里Guava版本需要在17以下

java.lang.ClassNotFoundException: com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_00
需要在项目中新建一个com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_00,内容与
com.thinkaurelius.titan.diskstorage.hbase.HBaseCompat1_0一致即可。

















猜你喜欢

转载自blog.csdn.net/wulinshishen/article/details/52955865
今日推荐